crimsonwarlock wrote: ↑29 Jan 2023
avasopht wrote: ↑29 Jan 2023
That's not entirely true.
What I stated was correct in the context I was stating it.
I should have been more clear.
There were quite a few things being said, but I'll just say that things are moving freakishly fast right now, and we're barely scratching the surface.
I was in disagreement with the notion they effectively take the median. Yes, this is true for the most common architectures with, let's call them, default settings.
If your regularization is working well and pushing more weights to zero, you should have an ANN with some separation to allow for specialization (similar to the overfitting that makes random forest ensembles work so well).
That still falls foul of just creating derivative predictions. But with enough layers, you can see things like dimensionality reduction emerge (that's something you'd ideally like to happen without having to explicitly create a dimensionality reduction topology).
Essentially, an ANN should be able to generalize the problem domain from the data to some degree. If dimensionality reduction occurs in a regular ANN then that means you've done a stellar job, and have a model that with generalize exceptionally well, but could also come up with good predictions that aren't even remotely in the dataset.
But there are ways to innovate in this particular space (something I'm interested in, and have found it to be a game changer with ensemble models).
crimsonwarlock wrote: ↑29 Jan 2023
avasopht wrote: ↑29 Jan 2023
There are plenty of ways to mix algorithms.
No, there are not, when we are talking about these systems. One of the biggest hurdles for ANN-technology is that it is very hard to integrate with other technology. That is the main reason that we didn't actually see any progress with, for example, autonomous vehicles.
I felt that AlphaGo and AlphaZero (ANNs are combined with Statistical Forward Planning) were good examples of integration good enough to learn to beat the best chess engines through self-play alone..
crimsonwarlock wrote: ↑29 Jan 2023
ChatGPT (and its predecessors GTP2 and GPT3), all the image generation stuff like Dal-e and Stable Diffusion, and everything else like that which came out in the last year, is based on Transformers. AlphaGo and AGzero are entirely different technology. And the biggy here is, again, that you can't simply integrate systems like AlphaGo and Tranformer-based systems.
Hmm ... I'm not too sure about that (of course, that depends on the sort of integration you mean).
Integrating systems like AlphaGo to be used in Transformer-based systems ... no.
Integrating Transformer-based systems to be used in Statistical Forward Planning based systems (like AlphaGo) ... yes (of course, not GPT3 with AlphaGo

).
I'm interested in the integration and being able to plug-and-play algorithms to just see what happens.
But it's hard for AI graduates without a firm foundation in software engineering to tackle integration (at least that's how it seems to me).