TLDR:
Secure connexion between tinybird and twenty using jwt when accessing
datasource from tinybird.
Solves:
https://github.com/twentyhq/private-issues/issues/73
In order to test:
1. Set ANALYTICS_ENABLED to true
2. Set TINYBIRD_JWT_TOKEN to the ADMIN token from the workspace
twenty_analytics_playground
3. Set TINYBIRD_JWT_TOKEN to the datasource or your admin token from the
workspace twenty_analytics_playground
4. Create a Webhook in twenty and set wich events it needs to track
5. Run twenty-worker in order to make the webhooks work.
6. Do your tasks in order to populate the data
7. Enter to settings> webhook>your webhook and the statistics section
should be displayed.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
Moving this command to the global command runner for 0.32 upgrade. This
should fix searchVector expression introduced in 0.31 to later handle
soft-deleted + search filter properly.
## Test
```
git co v0.31.0
-- reset DB
git co c--add-simplify-search-vector-expression-to-upgrade-0-32
-- migrate typeorm
-- upgrade 0.32 command
```
search is working and expression is correctly set. Soft deleted records
are not shown as before but it's now possible to override that behavior
via filters.
cc @ijreilly
## Context
Regression on custom index creation where indexFieldMetadatas were not
saved properly in the DB. This is because we recently changed save() to
upsert() in the indexMetadataService and upsert does not handle nesting
insert properly.
I'm suggesting another fix where we separate indexMetadata creation and
index migration creation in 2 different functions. Since the goal was to
be able to recreate the index after being deleted when we changed the
tsvector expression and indexMetadata was actually not deleted, we
didn't need to recreate that part (hence the upsert) and only needed to
run a migration to create the actual index in the workspace schema.
I've updated the different services and now only call
createIndexMigration when we update a search vector expression.
Note: this is also fixing the sync-metadata command when running on a
workspace with a custom object (including the seeded workspace which has
the 'rocket' custom object), failing due to the missing 'searchVector'
indexFieldMetadata
First step of #https://github.com/twentyhq/twenty/issues/3298.
Here we update the search endpoint to allow for a filter argument, which
we currently use in the relation pickers to restrict or exclude ids from
search.
In a future PR we will try to simplify the search logic in the FE
**Before:**
Only last 5 days where displayed on Developers Settings Webhook Usage
Graph.
**Now**
Added component where you can select the time range where you want to
view the webhook usage. To do better the styling and content depassing .
<img width="652" alt="Screenshot 2024-10-15 at 16 56 45"
src="https://github.com/user-attachments/assets/d06e7f4c-a689-49a0-8839-f015ce36bab9">
**In order to test**
1. Set ANALYTICS_ENABLED to true
2. Set TINYBIRD_TOKEN to your token from the workspace
twenty_analytics_playground
3. Write your client tinybird token in
SettingsDeveloppersWebhookDetail.tsx in line 93
4. Create a Webhook in twenty and set wich events it needs to track
5. Run twenty-worker in order to make the webhooks work.
6. Do your tasks in order to populate the data
7. Enter to settings> webhook>your webhook and the statistics section
should be displayed.
8. Select the desired time range in the dropdown
**To do list**
- Tooltip is truncated when accessing values at the right end of the
graph
- DateTicks needs to follow a more clear standard
- Update this PR with more representative images
When the user presses the cancel button, the server sends the following
response:
{"statusCode": 401, "message": "Unauthorized"}
Now, when the user clicks the cancel button, they are redirected to the
home page for login.
Related Issue
Fixes#7584
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
### Description
- We are using gql instead of strings to be able to see the graphql code
highlighted
### Demo
Fixes#7526
---------
Co-authored-by: gitstart-twenty <gitstart-twenty@users.noreply.github.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
Closes#7421
This pull request consolidates Redis connection parameters into a single
`REDIS_URL` environment variable across various configuration files and
code modules. The most important changes include updates to environment
variable files, Docker and Kubernetes configurations, and code modules
to utilize the new `REDIS_URL` format.
### Environment Variable Updates:
*
[`packages/twenty-docker/.env.example`](diffhunk://#diff-f4b5e7acc0dde630eafb2228390ca56bd56db0d183582be6433a9ee808088a4cL8-R8):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/.env.example`](diffhunk://#diff-c06e244338b33286ea90221635809785352a971de53f647ea65650124ae74ad2L53-R53):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
*
[`packages/twenty-server/.env.test`](diffhunk://#diff-def44a08e699c9deca2f72c9f87951de5d33d9ccf5621eab2f34978df8ad0954L16-R16):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
### Docker and Kubernetes Configuration Updates:
*
[`packages/twenty-docker/docker-compose.yml`](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL` in multiple
service definitions.
[[1]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28)
[[2]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL62-R61)
*
[`packages/twenty-docker/k8s/manifests/deployment-server.yaml`](diffhunk://#diff-91623ed4e8b2088947cfa9a5dad76b6013e8db0c150d84347a215fa9ad78bf04L44-R45):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/manifests/deployment-worker.yaml`](diffhunk://#diff-8532debd131ce168a2527fa6a9be6405792178576ee47e2eef0cd9f3ff4a8f8cL43-R44):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-server.tf`](diffhunk://#diff-b4c468660ab00bd03589d0d47502c477ca83f7d876ff196534880b02ba46dce0L64-R65):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-worker.tf`](diffhunk://#diff-e5d6a1f68d5391e2120aef6261f22f905aa1ce0a567e31b21a27d44f313eaecaL62-R63):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
### Code Module Updates:
*
[`packages/twenty-server/src/engine/core-modules/cache-storage/cache-storage.module-factory.ts`](diffhunk://#diff-06e787a7c8a48022d5909b5df9b8c6ca192521cf32f51d7f561cee937bed6678L23-R35):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/environment/environment-variables.ts`](diffhunk://#diff-26ce615693b053eda02aa48aa2e30400381a2588dcb08d4a9dc3b0bf5bdd6fe7L378-R384):
Added validation for `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module-factory.ts`](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
[[1]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6)
[[2]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bL35-R41)
### Documentation Updates:
*
[`packages/twenty-website/src/content/developers/self-hosting/self-hosting-var.mdx`](diffhunk://#diff-c4cc78a3ce18b6edb10f1aee8990271e1d2796a8c06c1c6ae3b68db8d52278a3L37-R37):
Updated documentation to reflect the change to `REDIS_URL`.
*
[`packages/twenty-website/src/content/developers/self-hosting/upgrade-guide.mdx`](diffhunk://#diff-c7d757829f6128e1f47aa1955bde561292dce558280511fab66281afa99042a5R102-R112):
Added upgrade instructions for the new `REDIS_URL` variable.
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
`isUnique` was passed to TypeORM's column creation API resulting in
double index creation because it's already done via the decorator and
then in `WorkspaceMigrationIndexFactory`
It would be interesting to move it at this field level in a later step,
which is why I also fixed `CompositeColumnActionFactory` to pass
isUnique on the correct columns, even though it's being ignored later on
## Context
This PR removes pg_graphql from the setup. It also updates the local
setup documentation accordingly.
Note: We removed local setup scripts to align with redis installation,
the setup should be much simpler since we don't rely on pg_graphql
anymore.
## Test
tested locally with docker + mac (brew) setup
By default, when custom fields are created, a searchVector field is
created based on the "name" field, which is also the label identifier by
default.
When this label identifier is updated, we want to update the
searchVector field to use this field as searchable field instead, if it
is of "searchable type" (today it is only possible to select a text or
number field as label identifier, while number fields are not
searchable).
## Context
WorkspaceIsUnique decorator allows us to create unique indexes on our
tables. Here Emails is a composite field containing a TEXT subfield. Due
to the fact that TEXT fields are non-nullable and have empty strings as
default values, adding a unique index on a TEXT field or any composite
containing TEXT as subfields will fail and throw if we try to create
more than 1 record that does not specify a value to the TEXT field.
This PR simply removes the index for the time being until we find a
solution
## Context
This PR removes workspace-query-runner/builder in preparation for fully
deprecating pg_graphql
next steps: Remove from the setup and make a command to remove comments
on schema/tables related to pg_graphql
Add support for indexes on composite fields and unicity constraint on
indexes
This pull request includes several changes across multiple files to
improve error handling, enforce unique constraints, and update database
migrations. The most important changes include updating error messages
for snack bars, adding a new command to enforce unique constraints, and
updating database migrations to include new fields and constraints.
### Error Handling Improvements:
*
[`packages/twenty-front/src/modules/error-handler/components/PromiseRejectionEffect.tsx`](diffhunk://#diff-e7dc05ced8e4730430f5c7fcd0c75b3aa723da438c26e0bef8130b614427dd9aL23-R23):
Updated error messages in `enqueueSnackBar` to use `error.message`
directly.
*
[`packages/twenty-front/src/modules/object-metadata/hooks/useFindManyObjectMetadataItems.ts`](diffhunk://#diff-74c126d6bc7a5ed6b63be994d298df6669058034bfbc367b11045f9f31a3abe6L44-R46):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useFindDuplicateRecords.ts`](diffhunk://#diff-af23a1d99639a66c251f87473e63e2b7bceaa4ee4f70fedfa0fcffe5c7d79181L56-R58):
Simplified error messages in `enqueueSnackBar`.
*
[`packages/twenty-front/src/modules/object-record/hooks/useHandleFindManyRecordsError.ts`](diffhunk://#diff-da04296cbe280202a1eaf6b1244a30490d4f400411bee139651172c59719088eL22-R24):
Simplified error messages in `enqueueSnackBar`.
### New Command for Unique Constraints:
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-enforce-unique-constraints.command.ts`](diffhunk://#diff-8337096c8c80dd2619a5ba691ae5145101f8ae0368a75192a050047e8c6ab7cbR1-R159):
Added a new command to enforce unique constraints on company domain
names and person emails.
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.command.ts`](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14):
Integrated the new `EnforceUniqueConstraintsCommand` into the upgrade
process.
[[1]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR13-R14)
[[2]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR31)
[[3]](diffhunk://#diff-20215e9981a53c7566e9cbff96715685125878f5bcb84fe461a7440f2e68f6fcR64-R68)
*
[`packages/twenty-server/src/database/commands/upgrade-version/0-31/0-31-upgrade-version.module.ts`](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7):
Registered the new `EnforceUniqueConstraintsCommand` in the module.
[[1]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R7)
[[2]](diffhunk://#diff-da52814efc674c25ed55645f8ee2561013641a407f88423e705dd6c77b405527R24)
### Database Migrations:
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368824-migrationDebt.ts`](diffhunk://#diff-c450aeae7bc0ef4416a0ade2dc613ca3f688629f35d2a32f90a09c3f494febdcR1-R53):
Added a migration to update the `relationMetadata_ondeleteaction_enum`
and set default values.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726757368825-addIsUniqueToIndexMetadata.ts`](diffhunk://#diff-8f1e14bd7f6835ec2c3bb39bcc51e3c318a3008d576a981e682f4c985e746fbfR1-R19):
Added a migration to include the `isUnique` field in `indexMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726762935841-addCompostiveColumnToIndexFieldMetadata.ts`](diffhunk://#diff-7c96b7276c7722d41ff31de23b2de4d6e09adfdc74815356ba63bc96a2669440R1-R19):
Added a migration to include the `compositeColumn` field in
`indexFieldMetadata`.
*
[`packages/twenty-server/src/database/typeorm/metadata/migrations/1726766871572-addWhereToIndexMetadata.ts`](diffhunk://#diff-26651295a975eb50e672dce0e4e274e861f66feb1b68105eee5a04df32796190R1-R14):
Added a migration to include the `indexWhereClause` field in
`indexMetadata`.
### GraphQL Exception Handling:
*
[`packages/twenty-server/src/engine/api/graphql/workspace-query-runner/utils/workspace-query-runner-graphql-api-exception-handler.util.ts`](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4):
Enhanced exception handling for `QueryFailedError` to provide more
specific error messages for unique constraint violations.
[[1]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R1-R4)
[[2]](diffhunk://#diff-58445eb362dc89e31107777d39b592d7842d2ab09a223012ccd055da325270a8R23-R59)
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-many-resolver.factory.ts`](diffhunk://#diff-233d58ab2333586dd45e46e33d4f07e04a4b8adde4a11a48e25d86985e5a7943L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
*
[`packages/twenty-server/src/engine/api/graphql/workspace-resolver-builder/factories/create-one-resolver.factory.ts`](diffhunk://#diff-68b803f0762c407f5d2d1f5f8d389655a60654a2dd2394a81318655dcd44dc43L58-R58):
Updated the `workspaceQueryRunnerGraphqlApiExceptionHandler` call to
include context.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
In this PR, I'm fixing part of the impact of soft deletion on optimistic
rendering.
## Backend Vision
1) Backend endpoints will not return soft deleted records (having
deletedAt set) by default. To get the softDeleted records, we will pass
a { withSoftDelete: true } additional param in the query.
2) Record relations will NEVER contain softDeleted relations
## Backend current state
Right now, we have the following behavior:
- if the query filters do not mention deletedAt, we don't return
softDeletedRecords
- if the query filters mention deletedAt, we take it into consideration.
Meaning that if we want to have the softDeleted records in any way we
need to do { or: [ deletedAt: NULL, deletedAt: NOT_NULL] }
## Optimistic rendering strategy
1) useDestroyOne/Many is triggering destroyOptimisticEffects (previously
deleteOptimisticEffects)
2) UseDeleteOne/Many and useRestoreOne/Many are actually triggering
updateOptimisticEffects (as they only update deletedAt field) AND we
need updateOptimisticEffects to take into account deletedAt (future
withSoftDelete: true) filter.
While rebuilding the new GraphQLAPI (without pg_graphql), we forgot to
include the FieldGetter logic. This logic will soon be moved at ORM
level but we will need to keep it there for now
The bug has many impacts such as the fileToken not being generated and
preventing users from loading files
Flush Specific Redis Keys After Upgrade Command :----
Changes Made :----
Updated the FlushCacheCommand to allow for selective flushing of keys
that match the pattern engine:*, rather than flushing the entire cache.
Added a new optional parameter to specify the cache keys pattern.
Ensured that the cache is flushed at the end of the upgrade command.
Code Changes :----
Modified FlushCacheCommand to include a method for flushing keys by
pattern.
Added a new function in CacheStorageService to handle the pattern-based
flushing.
---------
Co-authored-by: Weiko <corentin@twenty.com>
## Context
Because we can't create GIN indexes on primitive types and in this case
both tsvector and deletedAt (date), we had to create an index on the
tsvector only and had to make sure it wouldn't return results for
deleted records.
To handle this, we added a CASE in the generated expression to generate
an empty value if the row has been soft-deleted. This is a bit too
complex and won't allow search on softDeleted records in the future.
What we want to do is to make sure deleted records are not returned by
default by adding a WHERE clause in the search and still keep good
performances by also adding a BTREE index on deletedAt column.
When this was implemented, soft-deleted was not handled properly but now
typeorm query builder exclude soft deleted records by default which
means the WHERE clause is not necessary. As for the BTREE index on
deletedAt, this should be part on a broader effort to add it to all
tables and can be added in a later PR.
This PR simply updates the complex tsVectorExpression to only return the
expression regardless of the deletedAt column. Search won't return
soft-deleted records as explained above and perfs will be improved in an
upcoming version.
## Test
tested locally with reset db + created a new workspace. Since the
feature is not launched yet no more effort should be made.
<img width="1289" alt="Screenshot 2024-10-10 at 15 16 16"
src="https://github.com/user-attachments/assets/d9a6f06b-d738-4f1d-8053-2fa93bd8b4f8">
## Note
Some issues with optimistic rendering, we need to refresh once a record
has been deleted otherwise it will still be returned by the search
Expected behavior:
- workflows can be added and deleted. Only name field is editable
- versions and runs cannot be added nor deleted. No fields are editable
Added two new utils for those needs:
- `isReadOnlyObject` the similar logic between remote objects, versions
and runs
- `isFieldReadonlyFromObjectMetadataName` to easily block field edition
from object context
In this PR:
- remove deprecated EMAIL, PHONE, LINK field types (except for Zapier
package as there is another work ongoing)
- remove composite currency filter on currencyCode, actor filter on name
and workspaceMember as the UX is not great yet
In [this](https://github.com/twentyhq/twenty/pull/7522) and
[this](https://github.com/twentyhq/twenty/pull/7543) PR we introduced
the impossibility to save a field that would be non nullable but without
a default value.
The check is actually called on the input while the defaultValue is
added by the service on a "built" fieldMetadata to create or save. So
far all fields created from the app it currently fails as both
isNullable and defaultValue are undefined so falsy at that stage.
- Fixing seeds after introducing the requirement for non-nullable fields
to have a default value (https://github.com/twentyhq/twenty/pull/7522).
- Empty string needs to be considered a valid default value
#7346#7343#7342#7344
Before:
<img width="799" alt="Screenshot 2024-10-08 at 11 59 37"
src="https://github.com/user-attachments/assets/a1cd1714-41ed-4f96-85eb-2861e7a8b2c2">
Now:
In order to test:
1. Set ANALYTICS_ENABLED to true
2. Set TINYBIRD_TOKEN to your token from the workspace
_twenty_analytics_playground_
3. Write your client tinybird token in
SettingsDeveloppersWebhookDetail.tsx in line 93
4. Create a Webhook in twenty and set wich events it needs to track
5. Run twenty-worker in order to make the webhooks work.
6. Do your tasks in order to populate the data
7. Enter to settings> webhook>your webhook and the statistics section
should be displayed.
- update `send-email.workflow-action.ts` so it send email via the google
sdk
- remove useless `workflow-action.email.ts`
- add `send` authorization to google api scopes
- update the front workflow email step form to provide a
`connectedAccountId` from the available connected accounts
- update the permissions of connected accounts: ask users to reconnect
when selecting missing send permission
Fix syncedAt no longer been set on message sync.
Fix calendar data model:
- Add `syncedAt` to `CalendarChannelWorkspaceEntity`
- Move `recurringEventExternalId` from `CalendarEventWorkspaceEntity` to
`CalendarChannelEventAssociationWorkspaceEntity` since the id is
relative to one channel
Fix save queries on calendar sync after regression.
Enabling feature flags IsSearchEnabled and IsWorkspaceMigratedForSearch
at workspace creation to ensure workspaces have the searchVector fields
and indexes created.
For the feature to be enabled in the front-end we will also need
IsQueryRunnerTwentyORMEnabled to be enabled but that is an independent
topic.
Following this https://github.com/twentyhq/twenty/pull/7428 we now need
to fix existing workspaces thanks to this migration command.
This command will fetch all standard objects and their fields then
filter out tables that don't have that column OR objects that have an
existing fieldMetadata "name" of type TEXT and delete the rest.
