While I know there's going to be a lot of complications in this, given a quick search it seems like these GPUs are ~$2/hr, so $4000-4500 if you don't just have access to a cluster. I don't know how important the cluster is here, whether you need some minimal number of those for the training (and it would take more than 128x longer or not be possible on a single machine) or if a cluster of 128 GPUs is a bunch less efficient but faster. A 4B model feels like it'd be fine on one to two of those GPUs?
Also of course this is for one training run, if you need to experiment you'd need to do that more.
I like the way that people add “a friendly reminder” like they’re just jogging your memory of a well known fact.
Publishers have been fighting OA for an incredibly long time. They are not foisting this on people because it’s a new great scheme they’ve come up with, they have been pushed to do it.
That has nothing to do with this, those are things that should still be solved at a much higher level of abstraction. Tax the energy, pollution, waste - those have problems regardless of what caused them.
The point I am making is that the reason artefacts of highly automated production (even with minimal human labor required) will never become accessible for very low human labor, because all that automation has its own cost. We can externalise that as long as possible and defer the bill to somewhere or someone else, but it will have to be paid eventually.
> […] those are things that should still be solved at a much higher level of abstraction […]
I don't think that makes much sense. If a data center consumes all available electricity in a given municipality, it may provide AI services at a very low cost, but thereby makes the region uninhabitable. There is no way to "solve" this at a higher abstraction level. Or alternatively, consider a factory producing consumer goods, which emits toxic fumes; we can limit the amount of fumes the vicinity of the factory is exposed to by implementing very expensive filters—thus increasing the final price of the goods—or externalise all the negative effects—such as health risks in the population, ecological demise, and subsequently lower property values—to society, achieving a lower final price.
Currently, we often pick the latter option, because it usually has the better profit margin. I agree that it's a systemic issue that must be addressed holistically, but the actual solutions have to be implemented at all levels of the production chain. And this means the cost attached will have to be included in the price of all goods.
> The point I am making is that the reason artefacts of highly automated production (even with minimal human labor required) will never become accessible for very low human labor, because all that automation has its own cost
While I'm not sure I agree, this is not solved by tackling things at a low level and should be done at a higher level of abstraction - that's what they were saying.
> don't think that makes much sense. If a data center consumes all available electricity in a given municipality, it may provide AI services at a very low cost, but thereby makes the region uninhabitable.
If the data center was providing streaming services would you want to manage that differently? Imagine you had a data center that solved some user problem X, and another one that solves the same problem. Data center A uses AI, B does not but uses more power. Would you want to tax B less? Given what you've said so far I'd assume the answer is no - you'd want to tax that more because it's not really the AI part you care about, it's the power usage/emissions/local impact/externality X you want to avoid.
> I agree that it's a systemic issue that must be addressed holistically, but the actual solutions have to be implemented at all levels of the production chain.
Actually the more abstract sometimes the fewer places you have to deal with it. You don't have to figure out what cars everyone has, the specific MPG of each, driving patterns, how far your delivery driver went, whether they had other packages, etc - you can tax gasoline. This automatically flows through and avoids lots of wrangling about details and loopholes.
> Currently, we often pick the latter option, because it usually has the better profit margin.
Yes - and this drive makes it hard to manage when you put very precise rules around it. Tax AI and watch things rebrand as whatever falls just outside the limits of AI. See how products are built, deconstructed and remade exactly based on specific tariffs. Ford used to ship vans with windows and seats installed, then take them out again after they arrived!
> If AI makes a few people trillionaires while hollowing out the middle class, how do we keep the lights on?
Tax the thing you care about? You don't need to care really about the definition of AI or what an AI is or anything like that, you care that some people got trillions.
Tax "making an absolute shitton of money" or "being worth an insane amount". Taxing AI specifically means you're absolutely fucked if Altman turns out to not earn that much but someone who makes a specific connector to data centres is the richest person in the world. Is Nvidia an AI company? What is AI? *Who cares?* The point is to figure out some way as a society of continuing.
This is further info because I think it’s interesting rather that any sort of correction but fMRI doesn’t quite measure blood flow - at least not directly.
Oxygenated and deoxygenated blood have slightly different magnetic properties. So the fMRI is trying to detect from that how oxygenated the blood is, with the assumption that active areas are using more oxygen which causes a small dip then blood flow increases so then there’s an increase that follows over about 5-6 seconds. I don’t know if more advanced things are used now but when I messed about with it you’d measure the change then apply a 6s linear convolution to the signal to estimate activity.
There’s an interesting set of layers of assumptions in all this, and to me the idea that the mri part works at all seems like wild magic.
reply