NoDBA @NoDBA_

JSON is burning your CPU! Your API is slow. You blame the database. You blame the network. But the real bottleneck might be the language you are speaking. JSON is not a data format. It is a text string. Every time you send {"id": 12345}, your server pays a hidden 'Parse Tax.' Even with modern SIMD-optimized parsers, text processing faces architectural limits that binary formats do not. Here is the rigorous engineering breakdown: 1./ The CPU Cost (state machine vs. arithmetic) JSON (Text): To read the number 12345, the CPU receives raw bytes. Even the fastest parsers (like simdjson) must implement a State Machine: Scan for structural delimiters (: and ,). Check for escape sequences (\). the conversion: Loop through ASCII characters, subtracting '0', multiplying by powers of 10, and summing. This involves branch mispredictions and memory lookups. Protobuf (Binary): It sends a Varint (for small numbers) or Fixed-Width (for large ones). Fixed-Width (e.g., fixed32) => It is a raw memory copy (memcpy). Zero parsing. Varint => It reads 1 byte at a time, checking the "Most Significant Bit" (MSB) to see if the number continues. The Result => Decoding is effectively a few bitwise shifts and masks. For numeric-heavy workloads, this is mathematically guaranteed to be significantly faster than text parsing. 2./ The Bandwidth Cost (entropy vs. redundancy) JSON: [{"status": "active"}, {"status": "active"}] You are sending the key "status" repeatedly. Counter-argument => "But GZIP compression fixes this!" The Rebuttal => GZIP reduces the Network bytes, but it increases the CPU cost. Your server now has to Serialize JSON -> Compress -> Send. The receiver has to Decompress -> Parse JSON. You are burning CPU to compress redundant text that shouldn't have been there in the first place. Protobuf: It separates the Schema from the Data. The wire message replaces "status" with a Field ID (e.g., 1). [Tag: 1][Value: "active"]. This reduces the payload size before compression is even applied, saving CPU cycles on both ends. 3./ The Robustness Cost (schema-on-read) JSON is "Schema-on-Read." The receiver gets a blob. It hopes the ID is a number. Your code is full of runtime checks (if typeof(id) !== 'number'...) or you use a validation library (like Zod/Pydantic), which adds another layer of CPU overhead at runtime. Protobuf is "Schema-on-Write." It enforces a contract. While it doesn't catch logic errors, it guarantees Type Fidelity at the serialization boundary. You generally don't need expensive runtime validation libraries to check if an Integer is an Integer. In a nutshell - JSON is excellent for Public APIs (debuggable, easy). But for high-throughput Microservices, JSON is a tax. Switching to gRPC/Protobuf isn't magic. It is simply moving the complexity from Runtime (parsing text) to Compile Time (code generation). Happy Learning! Follow @techNmak for more insights.

@JaakkoDahlbacka Onko analysoitu, miksi Abballa lähti voimat käsistä? Toki jännitys ja melkein täysi aika, mutta entä painonvedon ero, kun olikin ehkä UFC:n ohjeilla, eikä entisillä totutuilla rutiineilla?

@ChristianLempa rustfs.com

@ChristianLempa @chainguard_dev is committed to support MinIO under its EmeritOSS program: chainguard.dev/unchained/fork…

MinIO's GitHub repo was just archived — officially no longer maintained as of Feb 12. If you're self-hosting S3-compatible storage, it's time to evaluate alternatives. What are you moving to? #selfhosted #MinIO #objectstorage

@JaakkoDahlbacka Joo, Tubessa huomattavasti parempi. Joku pakkaus tms. voi vaikuttaa.

@JaakkoDahlbacka Ootteko huomanneet huiman eron äänenlaadussa Youtube vs Spotify? Spotifyssä ääni on rosoinen.

If you're streaming data into DuckDB, INSERT statements become a bottleneck fast. DuckDB's Appender API bypasses the SQL layer entirely. No parsing, no query planning. You write directly to the columnar storage format, which means you can handle real-time ingestion without the usual speed/batch size trade-off. How it works: Stream rows through a low-level API. Data caches in batches before writing to disk. You're essentially using a binary protocol instead of SQL strings. Good for: -Kafka consumers or message queue ingestion -Log aggregation pipelines -IoT sensor data collection -Any scenario where data arrives continuously The trade-offs: It's order and type sensitive. You match columns exactly, no inference. One constraint violation fails the entire batch, no partial inserts. And you're writing to a single table per Appender instance. Available in C, C++, Go, Java, and Rust. For batch ETL or small datasets, regular INSERT is simpler and fine. But for streaming? This is the tool. Check out the docs: duckdb.org/docs/stable/da…

Snowflake ML just got faster, now with @nvidia’s cuML and cuDF libraries built in for native GPU acceleration. No rewrites, just results.

Snowflake @Snowflake

9 months ago

Your scikit-learn and pandas workflows can now easily scale up and run on GPUs in Snowflake! We are excited to announce that Snowflake ML now comes preinstalled with @nvidia's cuML and cuDF libraries, delivering native GPU acceleration for popular data science tools like

1 9 31 11K 5

Google just dropped the Gemini File Search API (RAG-as-a-Service). It allowed me to build a RAG chatbot in 31 min 🤯 No coding. Here’s how it works:

Semantic layer is not a protocol. I think that the data community needs to get this idea out of their heads that a higher order DSL that can express the semantics of data. MDS tried selling us that promise and failed. The real semantic knowledge of a business lives in unstructured docs and the minds of skilled domain experts. At best it's a giant, highly complex if-then-else of infinite depth. There is no way to codify this in a structured DSL. youtu.be/Oq009TByyWc?si…

If you’re still sending raw JSON into your LLMs, you’re burning tokens, latency, and budget! Try TOON (Token-Oriented Object Notation). Clear like YAML, compact like CSV: • 30–60% fewer tokens • Up to 50% lower costs • Shines for tabular data. Free and Open source 🧵↓

We have done a big disservice to metadata. Because software systems could only properly express and process structured metadata, we limited ourselves to table/column definitions with some (largely failed) attempts at expressing metrics in a structured format - yaml or json. All other metadata was meant for human consumption only. But that's merely scratching the surface of the vast amount of contextual information that accompanies data. This context is usually embedded in unstructured formats - PDFs, FAQs, internal docs, chats, meeting notes. Sure, you could surface this info in your favorite catalog tool but there was no way to actionize it or automate it. Until now. AI agents with tool calls open up a completely different way to process this information. For instance, a properly configured AI agent can read a user doc (PDF) to understand the definition of a metric (with all its complex nuances) and then execute a SQL query that incorporates this definition. Huge isn't even the word that describes the possibilities this unlocks.

when the bar goes down for access, the bar goes up for quality

Text to SQL conversion automation is still a big task and there are very few good open source models for this task. Let's breakdown this - >Text to SQL conversion models are basically nothing but encoder-decoder models with a multi-attention layer and a schema linking layer in between. > The encoder processes both the user query and the database schema and generates contextual embeddings ( relation-aware encoding ) >Through schema linking, tokens in the query are aligned with corresponding schema entities >The schema aware attention mechanism then allows the model to focus on relevant parts of the schema during decoding > The decoder sequentially produces SQL tokens ( constraint-based decoding ) Where do these models lack ? - >Most of the available models lack complex queries in training data itself and hence perform poorly on cross-domain or looped queries. > Language and query requirement is not always correct from a normal user. Even spelling mistakes leads to wrong entry and causes issues during retrieval hence prompting is an important part in this task. I personally worked on this in detail when I was making an end to end project, even made a synthetic data and tried training my own SLM but failed miserably and then went on to use an Open-source model. If you want to deep dive into this, i would recommend reading these research papers first - >LLM Enhanced Text-to-SQL Generation >Next-Generation Database Interfaces: >Text-to-SQL Parsing: Concepts and Methods >RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model

keyboard_arrow_left Previous keyboard_arrow_right Next

The PyData Amsterdam 2025 keynote “Minus Three Tier: Data Architecture Turned Upside Down� by Hannes Mühleisen is out now. youtube.com/watch?v=DxwDao…

Maintaining the right semantic context for text-2-sql is a delicate balance. Hence I keep thinking of it as semantic engineering. I think AI use cases will force semantics to surface in the data models themselves, instead of just being an add-on yaml/json on top of existing data models. Table names like "z_dep_cust1_new_backfill" will have to go out the door, if you plan to do any AI analytics on your data. No semantic models will help here. Same with table design. Normalize your data too much and AI will hallucinate the joins. Denormalize too much and your context window will be forever struggling for space.

Designing a chat experience is interesting because conversational nuances are very hard to get right. A simple Q and A loop soon starts feeling too rigid and robotic. To make it natural and human-like, a very fine-tuned workflow is needed. For ex., when do you decide to ask a clarifying question, how do you make it contextual, how often do you do this in a long running conversation. As with everything AI, a POC looks easy but it takes several weeks to get it right for real users.

Lots of database action this week. Yes, I have a new start-up @sydhtai with my PhD students (@lmwnshn + @17zhangw) using LLMs/ML to optimize almost everything in @PostgreSQL. The improvements are stunning! @datadictum posted a new article on our approach: theregister.com/2025/10/22/cmu…

DuckDB Labs is looking for a Customer Support Engineer to help customers set up and operate DuckDB. As Customer Support Engineer, you'll operate in the intersection of the DuckDB community, the DuckDB Labs customers, and the DuckDB core development team. duckdblabs.com/jobs/customer_…