Javi on DataData • AI • Cloud Engineering

Sujets populaires

↑↓ pour naviguer↵ pour ouvrirEsc pour fermer

Communauté

Ce dont parlent les ingénieurs de données, et mon avis pratique.

Je suis les discussions de la communauté data engineering et je rassemble les questions, les compromis et les problèmes de production qui méritent qu’on s’y attarde.

Cette page est une sélection, pas un flux automatique. Chaque sujet renvoie à la discussion d’origine et ajoute un résumé dans mes propres mots, pourquoi il compte et mon avis pratique.

Pouls de la communauté

Ce dont on parle dans la communauté data engineering.

Les sujets de la communauté sont d’abord rédigés en anglais. Chaque carte indique sa langue.

  • Source Microsoft Fabric CommunityEn anglais

    AI-assisted development in Fabric

    This is a vendor announcement, not a community thread: Microsoft’s FabCon Europe 2026 post on the Fabric Community blog. It describes agentic Copilot in notebooks, which turns a developer’s intent into governed, Fabric-native operations, and a data engineering agent in preview, built on the Osmos acquisition. Instead of step-by-step chat, developers describe an outcome and grant access to Fabric assets; the agent plans, executes and validates work over hours or days, creating notebooks, transforming data and updating schemas on Fabric Spark. Microsoft positions it for long-running efforts such as migrations, schema harmonization and data quality remediation. These are preview capabilities as described by Microsoft; production experience is still limited.

    Mon avis

    AI can accelerate boilerplate, exploration and repetitive implementation, but production data engineering still needs human ownership of architecture, security, schemas, performance and validation. Generated notebooks should be reviewed like any other code.

    • AI
    • LLM
    • Microsoft Fabric
    • Fabric Notebook

    À lire aussi

    Discuté le Ajouté le

    Source

  • Source RedditEn anglais

    CI/CD and Git deployment in Microsoft Fabric

    A team new to Git shared its design: workspaces per layer and environment, Dev workspaces connected to a dev branch, Test and Prod updated only by GitHub Actions using fabric-cicd, variable libraries for configuration, and SQL database projects for the Warehouse (which fabric-cicd did not deploy), with a manual review of the Warehouse diff before it runs. Report developers stay on deployment pipelines in separate workspaces. Replies challenged keeping Bronze in Prod only, since structural differences between environments tend to cause trouble later; suggested separating storage items from engineering items so feature branches can share Dev data; recommended dbt; and argued that branch policies and Git discipline matter more than the architecture. Since then, Microsoft’s August 2026 update has previewed Warehouse CI/CD with DacFx and an API for branched-workspace relationships.

    Mon avis

    Git should be the source of truth for code and deployable artifacts, while environment-specific configuration should remain externalized. Avoid treating a Fabric workspace as the source of truth. CI/CD should promote versioned artifacts into environments rather than depend on manual workspace edits.

    • CI/CD
    • Git
    • Microsoft Fabric
    • DevOps

    À lire aussi

    Autres discussions

    Discuté le Ajouté le

    Source

  • Source RedditEn anglais

    Delta Lake interoperability, schema and write behavior

    The question: when promoting Delta schema changes from Dev to Prod, rely on mergeSchema or overwriteSchema in the writer, or alter tables explicitly? The poster found switching mergeSchema on for one release and off again clunky. Answers differed: treat a table like a public API and never drop or retype columns (add a new column instead); keep mergeSchema on but control the columns explicitly before every write, except for Gold tables behind Direct Lake models; or route all writes through shared functions that enforce a YAML table definition. Related threads add the multi-engine side: delta-rs and Polars are fast and light for small workloads, but lag on Delta features (MERGE needed package upgrades in the Fabric runtime, Polars cannot write deletion vectors or V-Order), and one reply stated there are no plans to support Fabric-specific features in single-node libraries.

    Mon avis

    When multiple engines can write the same Delta table, the operational contract matters more than the library choice. Be explicit about schema ownership, supported protocol/table features, evolution rules and writer compatibility. Automatic schema merging is convenient, but it should not replace deliberate schema management in production.

    • Delta Lake
    • Schema Evolution
    • Fabric Lakehouse
    • Python

    À lire aussi

    Autres discussions

    Discuté le Ajouté le

    Source

  • Source RedditEn anglais

    Fabric architecture for a small team

    A poster shared a design for a small team: three repositories split by ownership (ingestion, analytics, reports), only Dev and Prod with no staging, Dev synced from main through Git integration, Prod promoted with deployment pipelines, a workspace pair per layer, metadata-driven Bronze, and shortcuts into a Gold workspace where Warehouse versus Lakehouse was still open. Feedback was mixed. Some warned about deployment pipelines and Warehouse deployment and suggested fabric-cicd, dbt or SQL database projects; the poster said deployment pipelines had worked so far. Others advised just starting and refactoring instead of designing everything up front, warned that capacity limits can push small teams toward larger SKUs, and offered different Gold choices depending on SQL versus Python skills and reporting performance.

    Mon avis

    Small teams benefit from fewer moving parts. The goal should not be to copy an enterprise architecture with dozens of workspaces and environments. Separate things where ownership, risk or deployment boundaries justify it; otherwise optimize for simplicity and maintainability.

    • Data Architecture
    • Microsoft Fabric
    • Fabric Workspace
    • Medallion Architecture

    À lire aussi

    Discuté le Ajouté le

    Source

  • Source RedditEn anglais

    Moving notebooks between Dev, Test and Production

    An engineer coming from Databricks Asset Bundles asked how Fabric handles environments. Replies agree on one workspace per environment but split on promotion. Several prefer the fabric-cicd Python library run from Azure DevOps or GitHub Actions, with Git integration only on per-developer feature workspaces, and describe deployment pipelines as tedious (rules per item, rebuilding the pipeline to change stages, unclear rollback); others say deployment pipelines work well for them, and one reply notes there is no polished equivalent yet. Environment-specific values belong in variable libraries, which one person found laborious to retrofit. A related thread on notebooks gives the practical version: replace the default Lakehouse and hard-coded ABFS paths with variables or paths resolved at run time, and expect a few manual fixes the first time new items reach Test or Prod.

    Mon avis

    Treat environment binding as configuration, not notebook logic. Notebooks should not require manual edits every time they move between Dev, Test and Production. Keep workspace-specific values externalized through variables, configuration or deployment tooling, and make environment promotion repeatable.

    • Fabric Notebook
    • Fabric Workspace
    • CI/CD
    • Git

    À lire aussi

    Autres discussions

    Discuté le Ajouté le

    Source

Question de la semaine

Comment participer

La communauté est une conversation. Voici les façons d’y prendre part, à mesure qu’elles deviennent disponibles.

  • Proposer un sujet

    Signalez-moi une discussion ou un problème de production qui mérite d’être approfondi.

  • Poser une question technique

    Les questions issues de vrais projets deviennent souvent des articles ou des sujets de la communauté.

  • Contribuer à un projet

    Aidez à construire les projets ouverts : code, documentation, exemples et tests.

  • Aider aux traductions

    Améliorez les versions française et espagnole des articles et des pages.

  • Signaler un problème

    Une erreur, un lien cassé ou un détail obsolète ? Dites-le-moi.

  • Partager ce que vous avez construit

    Vous avez construit quelque chose à partir d’un article ou d’un projet du site ? Partagez le résultat.

Les liens seront ajoutés ici dès que chaque canal sera en place.

Des lectures sur les sujets que je suis dans la communauté.

Articles

Projets

Sessions

Un sujet technique mérite d’être discuté ?

Si une discussion ici correspond à un problème sur lequel vous travaillez, ou si vous pensez qu’il manque quelque chose à mon avis, j’aimerais le savoir.

Proposer un sujet LinkedIn