Zos_Kia

If I understand these things correctly, the context window only affects how much text the model can “keep in mind” at any one time. It should not affect task performance outside of this factor.

Yeh, i did some looking up in the meantime and indeed you’re gonna have a context size issue. That’s why it’s only summarizing the last few thousand characters of the text, that’s the size of its attention.

There are some models fine-tuned to 8K tokens context window, some even to 16K like this Mistral brew. If you have a GPU with 8G of VRAM you should be able to run it, using one of the quantized versions (Q4 or Q5 should be fine). Summarizing should still be reasonably good.

If 16k isn’t enough for you then that’s probably not something you can perform locally. However you can still run a larger model privately in the cloud. Hugging face for example allows you to rent GPUs by the minute and run inference on them, it should just net you a few dollars. As far as i know this approach should still be compatible with Open WebUI.

There are not that many use cases where fine tuning a local model will yield significantly better task performance.

My advice would be to choose a model with a large context window and just throw in the prompt the whole text you want summarized (which is basically what a rag would do anyway).

If you like to write, I find that story boarding with stable diffusion is definitely an improvement. The quality of the images is what it is, but they can help you map out scenes and locations, and spot visual details and cues to include in your writing.

What’s hilarious about this is that they are so sensitive to being described. As a filthy liberal, you can call me weird is be like hell yeah strange-pilled bizarre-maxxing. Call me a communist I’ll be like you know it brother, stalin-with-laser-eyes.gif.

But call a right winger a weird fascist and they’re whining “well now let’s not resort to insults”. Why can’t they tolerate what they tend towards? Doesn’t sound very empowered to me.

Is that from one of those “rationalist” influencers?

Otherwise it’s just sparkling sedition

The anti left media blitz before during and after this election has been nothing short of disgusting. You’ve got people from a party actually founded by Waffen SS accusing the left of being anti-Semitic (because guess what, they’re not wholly approving of the genocide in Gaza). No fact check no push back just straight up “plutôt Hitler que front populaire”. They don’t even hide their shame anymore.

Holy shit it looks like it has dipped to 25. Is that common after market hours?

Yeah well, except those that don’t, I guess