airbyte
约 2 分钟阅读
- airbytehq/airbyteGitHubairbytehq/airbyteairbytehq/airbyte · MIT+ELv2, Java+Python+TypeScript · ETL · 实现用到 temporal 进行调度 · Source · APIs - Github, Gitlab, Notion笔记：airbyte
- MIT+ELv2, Java+Python+TypeScript
- ETL
- 实现用到 temporal 进行调度
- Source
- APIs - Github, Gitlab, Notion
- Database - PostgreSQL, MySQL, ClickHouse, CockroachDB, MongoDB
- Files
- Format - CSV, JSON, jsonl, excel, parquet, feather
- Provider - HTTP, S3, SFTP, SSH SCP
- Queue - Kafka
- RESTful
- Destination
- Kafka, MQTT, RabbitMQ, Pulsar, Redis
- S3, CSV, JSON, SFTP
- MySQL, PostgreSQL, Cassandra, ElasticSearch, MongoDB, MeiliSearch
- CDC
- PostgreSQL - wal2json, pgoutput
- MySQL
- MSSQL
- 参考
- 开源版本 API 目前无认证
Docker
- docker-compose.yamlGitHub 文件airbytehq/airbyte/docker-compose.yaml
- airbyte/scheduler
- airbyte/webapp
- airbyte/server
- airbyte/temporalGitHub 文件airbytehq/airbyte/airbyte-temporal
- 基于 temporal 官方脚本
- 重新构建添加 M1 支持
- temporal/dynamicconfig/development.yamlGitHub 文件airbytehq/airbyte/temporal/dynamicconfig/development.yaml
- limit.blobSize.warn=10MB - 默认 512KB
- limit.blobSize.error=15MB - 默认 2MB
- airbyte/worker
- airbyte/db - PostgreSQL
- airbyte/bootloader
- airbyte/init
- docker-compose.debug.yamlGitHub 文件airbytehq/airbyte/docker-compose.debug.yaml
- temporalio/web
Kubernetes
- Known Issues
- Helm charts/airbyteGitHub 文件airbytehq/airbyte/charts/airbyteairbytehq/airbyte · MIT+ELv2, Java+Python+TypeScript · ETL · 实现用到 temporal 进行调度 · Source · APIs - Github, Gitlab, Notion笔记：airbyte
- charts
- bitnami-common
- postgresql
- minio
- temporal - 内置
- charts
- Kustomization resourcesGitHub 文件airbytehq/airbyte/kube
Notes
- Scheduler
- API -> Temporal
- 参考
| field | desc |
|---|---|
_airbyte_ab_id | uuid |
_airbyte_emitted_at | timestamp-millis |
_airbyte_additional_properties | map of string |
_airbyte_data | 实际数据 |
_airbyte_normalized_at | |
_ab_cdc_updated_at | |
_ab_cdc_deleted_at | |
_ab_cdc_lsn | PostgreSQL, MSSQL CDC |
_ab_cdc_log_file | MySQL CDC |
_ab_cdc_log_pos | MySQL CDC |
Integrations
FAQ
When to use CDC
- 希望获取到删除信息
- 数据量大
- 表里没有增量同步信息 - 例如: updated_at
关联信息
反向链接、本文链接的其他页面和外部资料。
参考资料
GitHub9 条
- airbytehq/airbyte/airbyte-temporalgithub.com/airbytehq/airbyte/tree/master/airbyte-temporal
- airbytehq/airbytegithub.com/airbytehq/airbyte
- airbytehq/airbyte/airbyte-api/src/main/openapi/config.yamlgithub.com/airbytehq/airbyte/blob/master/airbyte-api/src/main/openapi/config.yaml
- airbytehq/airbyte/charts/airbytegithub.com/airbytehq/airbyte/tree/master/charts/airbyte
- airbytehq/airbyte/docker-compose.debug.yamlgithub.com/airbytehq/airbyte/blob/master/docker-compose.debug.yaml
- airbytehq/airbyte/docker-compose.yamlgithub.com/airbytehq/airbyte/blob/master/docker-compose.yaml
- airbytehq/airbyte/kubegithub.com/airbytehq/airbyte/tree/master/kube
- airbytehq/airbyte/temporal/dynamicconfig/development.yamlgithub.com/airbytehq/airbyte/blob/master/temporal/dynamicconfig/development.yaml
- mrin9/RapiDocgithub.com/mrin9/RapiDoc
其他外链8 条
- airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.htmlairbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html
- demo.airbyte.iodemo.airbyte.io/
- docs.airbyte.com/deploying-airbyte/on-kubernetesdocs.airbyte.com/deploying-airbyte/on-kubernetes
- docs.airbyte.com/integrations/sourcesdocs.airbyte.com/integrations/sources
- docs.airbyte.com/project-overview/changelogdocs.airbyte.com/project-overview/changelog
- docs.airbyte.com/project-overview/licenses/examplesdocs.airbyte.com/project-overview/licenses/examples
- docs.airbyte.com/understanding-airbyte/airbyte-specificationdocs.airbyte.com/understanding-airbyte/airbyte-specification
- docs.airbyte.com/understanding-airbyte/high-level-viewdocs.airbyte.com/understanding-airbyte/high-level-view