Commit Graph

62 Commits

Author SHA1 Message Date
Félix Malfait
736635a94b
Begin moving to postgres spilo + adding pgvector (#8309)
We will remove the `twenty-postgres` image that was used for local
development and only use `twenty-postgres-pilo` (which we use in prod),
bringing the development environment closer to prod and avoiding having
to maintain 2 images.


Instead of provisioning the super user after the db initialization, we
directly rely on the superuser provided by Spilo for simplicity. We also
introduce a change that tries to create the right database (`default` or
`test`) based on the context.
  

How to test:
```
docker build -t twentycrm/twenty-postgres-spilo:latest -f ./packages/twenty-docker/twenty-postgres-spilo/Dockerfile .
docker images --no-trunc | grep twenty-postgres-spilo
postgres-on-docker:
	docker run \
	--name twenty_pg \
	-e PGUSER_SUPERUSER=twenty \
	-e PGPASSWORD_SUPERUSER=twenty \
	-e ALLOW_NOSSL=true \
	-v twenty_db_data:/home/postgres/pgdata \
	-p 5432:5432 \
	REPLACE_WITH_IMAGE_ID
```
2024-11-15 09:38:30 +01:00
ZiaCodes
57d9b8e8b4
feat: generate secret function and replaced few instances (#7810)
This PR fixes #4588

---------

Co-authored-by: Félix Malfait <felix@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
2024-10-30 12:07:11 +01:00
Thomas Mol
ba2ee0da72
Refactor Redis connections to use Redis URL - closes #7421 (#7736)
Closes #7421

This pull request consolidates Redis connection parameters into a single
`REDIS_URL` environment variable across various configuration files and
code modules. The most important changes include updates to environment
variable files, Docker and Kubernetes configurations, and code modules
to utilize the new `REDIS_URL` format.

### Environment Variable Updates:
*
[`packages/twenty-docker/.env.example`](diffhunk://#diff-f4b5e7acc0dde630eafb2228390ca56bd56db0d183582be6433a9ee808088a4cL8-R8):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/.env.example`](diffhunk://#diff-c06e244338b33286ea90221635809785352a971de53f647ea65650124ae74ad2L53-R53):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
*
[`packages/twenty-server/.env.test`](diffhunk://#diff-def44a08e699c9deca2f72c9f87951de5d33d9ccf5621eab2f34978df8ad0954L16-R16):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.

### Docker and Kubernetes Configuration Updates:
*
[`packages/twenty-docker/docker-compose.yml`](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL` in multiple
service definitions.
[[1]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28)
[[2]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL62-R61)
*
[`packages/twenty-docker/k8s/manifests/deployment-server.yaml`](diffhunk://#diff-91623ed4e8b2088947cfa9a5dad76b6013e8db0c150d84347a215fa9ad78bf04L44-R45):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/manifests/deployment-worker.yaml`](diffhunk://#diff-8532debd131ce168a2527fa6a9be6405792178576ee47e2eef0cd9f3ff4a8f8cL43-R44):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-server.tf`](diffhunk://#diff-b4c468660ab00bd03589d0d47502c477ca83f7d876ff196534880b02ba46dce0L64-R65):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-worker.tf`](diffhunk://#diff-e5d6a1f68d5391e2120aef6261f22f905aa1ce0a567e31b21a27d44f313eaecaL62-R63):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.

### Code Module Updates:
*
[`packages/twenty-server/src/engine/core-modules/cache-storage/cache-storage.module-factory.ts`](diffhunk://#diff-06e787a7c8a48022d5909b5df9b8c6ca192521cf32f51d7f561cee937bed6678L23-R35):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/environment/environment-variables.ts`](diffhunk://#diff-26ce615693b053eda02aa48aa2e30400381a2588dcb08d4a9dc3b0bf5bdd6fe7L378-R384):
Added validation for `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module-factory.ts`](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
[[1]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6)
[[2]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bL35-R41)

### Documentation Updates:
*
[`packages/twenty-website/src/content/developers/self-hosting/self-hosting-var.mdx`](diffhunk://#diff-c4cc78a3ce18b6edb10f1aee8990271e1d2796a8c06c1c6ae3b68db8d52278a3L37-R37):
Updated documentation to reflect the change to `REDIS_URL`.
*
[`packages/twenty-website/src/content/developers/self-hosting/upgrade-guide.mdx`](diffhunk://#diff-c7d757829f6128e1f47aa1955bde561292dce558280511fab66281afa99042a5R102-R112):
Added upgrade instructions for the new `REDIS_URL` variable.

---------

Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
2024-10-16 17:17:44 +02:00
Charles Bochet
1b99a05dec Put back twenty-website dockerfile 2024-10-15 18:36:10 +02:00
Weiko
7e808cf529
Remove pgGraphql from setup (#7716)
## Context
This PR removes pg_graphql from the setup. It also updates the local
setup documentation accordingly.

Note: We removed local setup scripts to align with redis installation,
the setup should be much simpler since we don't rely on pg_graphql
anymore.

## Test
tested locally with docker + mac (brew) setup
2024-10-15 16:59:44 +02:00
BOHEUS
a7a4052109
Docker cleanup (#7655)
Related to #7648 

@charlesBochet I've deleted only dev related commands in Makefile, if
it's wrong, I'll change it ASAP
2024-10-13 20:40:27 +02:00
Ciara Hatcher
db9ec58f5d
Updating terraform and k8s files adding redis... (#7425)
Also updated the way secrets are generated with Terraform and some code
cleanup
2024-10-07 10:23:42 +02:00
Charles Bochet
e4959ad534
Add 0.30 release notes (#7300)
In this PR:
- update your environment variables to default `CACHE_STORAGE_TYPE` to
`redis` and `MESSAGE_QUEUE_TYPE` to `bull-mq`
- add redis container to our default docker-compose
- add `REDIS_HOST` and `REDIS_PORT` to docker-compose yaml
- add upgrade instructions
2024-09-27 19:10:26 +02:00
Quentin G.
d1871a4a64
feat: replace ts-node with tsx and pre-install it globaly in docker (#7027)
Implement https://github.com/twentyhq/twenty/issues/5976 suggestion.
It leads to faster boot time since we don't have to install
dependencies.

It should also solve https://github.com/twentyhq/twenty/issues/4772 as
we don't need to run ressource-heavy `npm install` (the mentioned
migration step does not seem to exist anymore anyway.
2024-09-14 11:05:01 +02:00
Quentin G.
b0bc839a49
feat: improve self hosting documentation (#7024)
This PR aims to resolve common misunderstanding while deploying Twenty
with Docker compose.

I've made the documentation clearer, and more detailed.

This should solve https://github.com/twentyhq/twenty/issues/5184, #6140
and similar issues
2024-09-14 11:03:28 +02:00
Ciara Hatcher
c3bf94e4cc
Lumosviridi v20 kubernetes updates (#6356)
Updates for v20+ and misc terraform bug fixes. Also refactored to use
terraform variables instead of locals which helps with readability and
ease of use for new users.

Terraform validation is currently passing:
![Screenshot 2024-07-21 at 13 18
37](https://github.com/user-attachments/assets/02aadc2d-d3f6-4e8b-9315-64e25191d9e6)

Additionally added [terraform-docs](https://terraform-docs.io/) to
generate a more helpful README for terraform specific configuration.

Raw K8s manifests were updated with changes for v20+ as well.

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2024-08-08 13:55:45 +02:00
Sandab Gharti GC
7e80cd5366
mod: entrypoint for docker to setup db even in server with less memory (#5815)
Hello team,

- Modified File: `packages/twenty-docker/twenty/entrypoint.sh`
- Reason: The current `entrypoint.sh` works fine for server with RAM
more than 8GB. But the server with less RAM will give error while
running `scripts/setup-db.ts`.

![setup-db-failed](https://github.com/twentyhq/twenty/assets/52379037/5e27240a-23ef-4d52-93bb-337e1a004feb)

- My commit: It will set `NODE_OPTIONS="--max-old-space-size=8192"` in
docker entrypoint which guarantee to run the script successfully.

---------

Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
2024-07-19 14:54:00 +02:00
Ciara Hatcher
43016db801
Support for Kubernetes via Terraform and Manifests (#5721)
# Support for Kubernetes via Terraform and Manifests

Adding basic support for the following Kubernetes resources

- persistent volume
  - server
  - database
- persistent volume claim
  - server
  - database
- deployment
  - server
  - database
- ingress
  - server
- service
  - server
  - database
- secret
  - server

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2024-07-10 13:54:27 +02:00
martmull
cff8561597
Upgrade pg graphql version to 1.5.6 (#5937)
- update `pg_graphql` version doc
- update `pg_graphql` version to 1.5.6
2024-06-18 17:34:16 +02:00
Jérémy M
04edf2bf7b
feat: add resolve absolute path util (#5836)
Add a new util called `resolveAbsolutePath` to allow providing absolute
path for environment variable like `STORAGE_LOCAL_PATH`.
If the path in the env start with `/` we'll not prefix it with
`process.cwd()`.

Also we're using a static path for the old `db_initialized` file now
named `db_status` and stop using the env variable for this file as this
one shouldn't ne stored in the `STORAGE_LOCAL_PATH`.

Fix #4794

---------

Co-authored-by: Quentin Galliano <qgalliano@gmail.com>
2024-06-12 21:17:31 +02:00
Charles Bochet
a57e251208
Fix docs build in CI (#5826) 2024-06-11 19:06:37 +02:00
bosiraphael
f166171a1c
5531 update gmail full sync to v2 (#5674)
Closes #5531

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-05-31 13:29:58 +02:00
Quentin G
c1740e3395
Feat/add postgres spilo (#5049)
This is a new Docker image that extends the Zalando Spilo image in order
to install the extensions needed by Twenty : pg_graphql, wrappers,
mysql_fdw
2024-04-29 17:09:41 +02:00
Thaïs
d3170fc1ea
fix: fix root start script (#5032)
Fixes #5022

See https://nx.dev/recipes/running-tasks/root-level-scripts#setup
2024-04-19 18:28:02 +02:00
Thaïs
7065495223
fix: attempt to fix Dockerfile front build (#5020) 2024-04-18 11:24:39 +02:00
Thaïs
9f83cc1426
refactor: move @/ui/display/icon to twenty-ui (#4820)
Split from https://github.com/twentyhq/twenty/pull/4518

Part of https://github.com/twentyhq/twenty/issues/4766
2024-04-12 15:30:48 +02:00
Quentin G
bf60227d67
feat: add SENTRY_RELEASE env (#4912)
Add support for a new SENTRY_RELEASE and SENTRY_ENVIRONMENT env.
It is optional and allows to init sentry with a Release version and an
env (used internally at Twenty).
Docker image have been updated do intergrate the new env as an Argument
2024-04-11 16:53:15 +02:00
Quentin G
e48960afbe
chore: remove old deployments (#4922)
We migrated all Twenty's env to the new Twenty docker image. We can now
remove old deployments of front and back.
This PR takes care of that!
2024-04-11 16:35:11 +02:00
Félix Malfait
7a34dc4910
Simplify docs and remove Docker local setup (#4783)
Having 2 different dev setups caused confusion, let's remove the Docker
local setup and recommend people install yarn locally.

Also simplified some docs by merging pages together, the recommend
self-hosting option is now the docker-compose / to adapt the
docker-compose.
2024-04-03 16:38:28 +02:00
Quentin G
35fb77d9a8
feat: reinforce one liner experience (#4688)
* feat: add more dependencies check, randomize postgres admin password, tail logs of server container

* feat: improve retro compatibility

* feat: comment POSTGRES_ADMIN_PASSWORD as it will be generated by the one liner
2024-03-29 15:25:41 +01:00
Quentin G
d6de380e02
feat: add one liner install command (#4613)
* feat: add one liner

* fix: interactive issue & add support for both linux & mac

* feat: move quick start documentation

* feat: catch errors

* feat: check if directory exists

* feat: default to yes for prompt

* feat: open in browser

* fix: format

* feat: do not expose STORAGE_LOCAL_PATH env but handle the case where it would be set

* fix: db reset command wasn't working out of the box

* Update install.sh

Co-authored-by: Darek Desu <4459421+darekdesu@users.noreply.github.com>

* feat: harden the whole UX with one-liner

* fix: small logical order adjustment

* Update packages/twenty-docs/docs/start/self-hosting/docker-compose.mdx

---------

Co-authored-by: Darek Desu <4459421+darekdesu@users.noreply.github.com>
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
2024-03-27 21:28:03 +01:00
Quentin G
90ce7709dd
fix: update docker-compose database volumes (#4677) 2024-03-27 08:47:58 +01:00
Darek Desu
22d17d855c
File token chores (#4664)
* Missing file token chores

* Make whole idea folder ignored
2024-03-26 13:42:09 +01:00
Quentin G
ef8867e552
chore: debug Twenty dockerfile failing to build on Github Actions (#4658)
* chore: debug Twenty dockerfile failing to build on Github Actions

* fix: remove commented code
2024-03-26 12:32:53 +01:00
Darek Desu
0549313c43
Update .env.example (#4661)
Added missing FILE_TOKEN_SECRET variable
2024-03-26 12:31:57 +01:00
martmull
e2af5b8628
4525 upgrade pg graphql (#4646)
* TO remove: add multiple workspace with command

* TO remove: update build script

* Update script and add doc

* TO remove: add more seed workspaces

* Build pg_graphql files for 1.5.1

* Build pg_graphql files for 1.5.1 macos arm

* Remove 600 seed workspaces

* Build pg_graphql files for 1.5.1 macos intel
2024-03-25 15:08:17 +01:00
Quentin G
aee6d49ea9
feat: add a docker-compose file for production (#4609)
* feat: add a docker-compose file for production

* fix: remove unused user filed

* fix: do not provide default token secrets
2024-03-22 09:16:39 +01:00
Quentin G
1aa48d3bf7
feat: merge front and server dockerfiles and optimize build (#4589)
* feat: merge front and server dockerfiles and optimize build

* fix: update image label

* fix: bring back support for REACT_APP_SERVER_BASE_URL injection at runtime

* fix: remove old entries & add nx cache in dockerignore

* feat: generate frontend config at runtime using Nest

* fix: format and filename

* feat: use the EnvironmentService and leave default blank

* feat: add support for DB migrations
2024-03-21 19:22:21 +01:00
Félix Malfait
b6e8bb1a6c
Delete auth/file front variables (#4455) 2024-03-19 22:19:40 +01:00
Charles Bochet
eb07b373a7 Fix storage s3 endpoint not being optional 2024-03-15 23:08:30 +01:00
Charles Bochet
feebc45d31
Fix storybook tests on IconPicker (#4510) 2024-03-15 21:47:07 +01:00
Charles Bochet
5e5ae0b2ca Fix server container build 2024-03-15 21:38:40 +01:00
Charles Bochet
e0ae12ffa1
Fix server deploy (#4509)
* Fix server deployment broken by nx

* Fix server deployment broken by nx

* Fix server deployment broken by nx

* Fix

* Fix

* Fix

* Fix
2024-03-15 21:15:14 +01:00
Charles Bochet
4eae3b8290
Fix front production container build (#4478)
* Fix deploy main

* Fix deploy main
2024-03-14 16:13:27 +01:00
Charles Bochet
aa13b8338d Fix website build 2024-02-25 22:47:07 +01:00
Charles Bochet
fa9edad311 Fix website build 2024-02-25 22:38:12 +01:00
Charles Bochet
88990144cf
Fix website build (#3986)
* Fix website build

* Try fix

* Try fix
2024-02-15 11:03:29 +01:00
Charles Bochet
9777c5fbce
Fix website build (#3983) 2024-02-14 21:28:26 +01:00
Thaïs
e011ecbd6f
POC: generate twenty-server package.json with Nx (#3654)
* POC: generate twenty-server package.json with Nx

* Re-add passport

* Fix instal

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2024-02-13 22:16:21 +01:00
Thaïs
a654205dbc
chore: set up twenty-emails config so build isn't needed in development (#3619)
* chore: set up twenty-emails config so build isn't needed in development

* fix: fix script dependency

* chore: use @vitejs/plugin-react-swc

* Remove useless dependancy

* Fix typing

* chore: use baseUrl in twenty-emails

* chore: fix docker server prod build

* refactor: optimize Docker file and tsconfig

* fix: fix WORKDIR in docker

---------

Co-authored-by: martmull <martmull@hotmail.fr>
2024-01-29 06:17:12 -03:00
Charles Bochet
3f0743493b
Fix Continuous Deployment script after adding twenty-emails package (#3589)
* Fix cd

* Fix server

* Fix server

* Fix server

* Fix server

* Fix

* Fix docs

* Fix
2024-01-23 12:30:03 +01:00
Charles Bochet
762a56782c Fix build server 2024-01-23 10:46:36 +01:00
Charles Bochet
a7265fa3b4
Remove flag relation select (#3588)
* Remove feature flag on relation and select

* Move packages back to twenty-server to enable smaller build without using nx

* Fix package.json
2024-01-23 09:59:00 +01:00
Charles Bochet
03bf597301 Re-add docker-compose up command 2024-01-13 11:02:00 +01:00
Charles Bochet
6672b04733
Fix docker-compose commands (#3403) 2024-01-12 14:31:47 +01:00