Hugging Face's logo
Weronika Stryj's picture
11 30

Weronika Stryj

privategeek24
pcuenq's profile picture venkatace's profile picture nautilie7's profile picture
·

AI & ML interests

Text classification, Sentiment Analysis, Finance Risk analysis, Text generation, Question answearing, Optimization LLM, Cyber risk analysis, Speech to text

Recent Activity

liked a Space about 2 months ago
sentence-transformers/quantized-retrieval
reacted to singhsidhukuldeep's post with 👀 4 months ago
While Google's Transformer might have introduced "Attention is all you need," Microsoft and Tsinghua University are here with the DIFF Transformer, stating, "Sparse-Attention is all you need." The DIFF Transformer outperforms traditional Transformers in scaling properties, requiring only about 65% of the model size or training tokens to achieve comparable performance. The secret sauce? A differential attention mechanism that amplifies focus on relevant context while canceling out noise, leading to sparser and more effective attention patterns. How? - It uses two separate softmax attention maps and subtracts them. - It employs a learnable scalar λ for balancing the attention maps. - It implements GroupNorm for each attention head independently. - It is compatible with FlashAttention for efficient computation. What do you get? - Superior long-context modeling (up to 64K tokens). - Enhanced key information retrieval. - Reduced hallucination in question-answering and summarization tasks. - More robust in-context learning, less affected by prompt order. - Mitigation of activation outliers, opening doors for efficient quantization. Extensive experiments show DIFF Transformer's advantages across various tasks and model sizes, from 830M to 13.1B parameters. This innovative architecture could be a game-changer for the next generation of LLMs. What are your thoughts on DIFF Transformer's potential impact?
reacted to davidberenstein1957's post with 👍 4 months ago
Don't use an LLM when you can use a much cheaper model. The problem is that no one tells you how to actually do it. Just picking a pre-trained model (e.g., BERT) and throwing it at your problem won't work! If you want a small model to perform well on your problem, you need to fine-tune it. And to fine-tune it, you need data. The good news is that you don't need a lot of data but instead high-quality data for your specific problem. In the latest livestream, I showed you guys how to get started with Argilla on the Hub! Hope to see you at the next one. https://www.youtube.com/watch?v=BEe7shiG3rY
View all activity

Organizations

None yet

privategeek24's activity

liked a Space about 2 months ago
liked 4 models 4 months ago
liked 6 models 5 months ago
liked a Space 5 months ago
liked 2 models 5 months ago
liked a Space 5 months ago
liked 5 models 5 months ago