You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Ready for payment][$250] Add some basic runtime type validation in Onyx #39852
It is possible to queue incorrect Onyx updates from Auth. Twice now, we have seen an array used in place of an object in an Onyx update in Auth, which when merged completely removes the object and replaces it with an array.
Solution
While we recognize that there a number of more elegant solutions to this (compile time type validation of Onyx updates in the back-end being a prime candidate), we should apply a simple solution to prevent simple mistakes from having catastrophic effects, as we have seen occur in production twice already.
What this means is - let's update Onyx such that:
it will not apply any updates where we try to merge an array into an object, or vice-versa
it will log an alert if we do try to do that (except on dev, of course)
@ShridharGoel Wouldn't we need to check if it's an Array for the existingValue as well? I am assuming we're using the existing block with OnyxUtils.get(key).then(existingValue.
Please re-state the problem that we are trying to solve in this issue.
We want to:
It will not apply any updates where we try to merge an array into an object, or vice-versa
It will log an alert if we do try to do that (except on dev, of course)
What is the root cause of that problem?
We don't have such functionality in Onyx yet
What changes do you think we should make in order to solve the problem?
A. Updates for Onyx.merge:
First there're a few considerations:
There're 3 cases possible for a change:
Nullish (null, undefined)
Object
Array
If existing value is Array, it can be used with Nullish/Array change, but not Object change. If existing value is Object, it can be used with Nullish/Object change, but not Array change.
So just checking Array.isArray and compare both is not enough, we need to check the Nullish case as well.
We can define a method isUpdateCompatibleWithExistingValue to validate that the update and the existing value are compatible:
function isUpdateCompatibleWithExistingValue(value, existingValue): boolean {
if (existingValue && value && Array.isArray(existingValue) !== Array.isArray(value)) {
return false;
}
return true;
}
The changes are batched into an array, we should only remove the invalid changes, not discard the full batch just because one of the update happens to be of the wrong type.
With the above considerations, here're the pseudo code of the changes we can make in Onyx here to filter out the invalid change, Log an alert, and check the change type compatibility properly:
const validChanges = mergeQueue[key].filter((change) => isUpdateCompatibleWithExistingValue(change, existingValue));
if (validChanges.length !== mergeQueue[key].length) {
Logger.logAlert(`A warning occurred while applying merge for key: ${key}, Warning: Trying to merge array to object or vice versa`);
}
if (!validChanges.length) {
return undefined;
}
Then below that, when we use mergeQueue[key] like in here, we should change to use validChanges instead.
B. Updates for Onyx.set
In here, we should get the existing value, then check if the new update is compatible with the existing value, if not, early return.
(we can alternatively get the existing value from cache.getValue so it can be synchronous)
OnyxUtils.get(key).then(existingValue => {
if (!isUpdateCompatibleWithExistingValue(value, existingValue)) {
Logger.logAlert(`A warning occurred while applying set for key: ${key}, Warning: Trying to set array to object or vice versa`);
return Promise.resolve();
}
// now we do the Onyx.set as now
C. Updates for Onyx.mergeCollection
In here, modify so getCachedCollection accepts an optional collectionMemberKeys param, if it's passed in, we'll use it, if not, we'll use our own existing collectionMemberKeyshere
In here, get the collection value of existingKeys by using getCachedCollection and pass existingKeys as collectionMemberKeys
In here, we check that mergedCollection[key] is compatible with the existing value from existingKeys above, if they are not compatible, Log a warning by using Logger.logAlert, and do not set to obj (so that update will be ignored)
What alternative solutions did you explore? (Optional)
For A., an alternative is to check update compatibility right in here before we push the changes to the mergeQueue. We can get the existing value by using cache.getValue and check the compatibility using isUpdateCompatibleWithExistingValue, if they are not compatible, log a warning using Logger.logAlert and early return, so the changes will never end up in the merge queue.
changed the title [-][$250] Add some basic runtime type validation in Onyx[/-][+][Ready for payment][$250] Add some basic runtime type validation in Onyx[/+]on Jun 24, 2024
Activity
[-]Add some basic runtime type validation in Onyx[/-][+][$250] Add some basic runtime type validation in Onyx[/+]melvin-bot commentedon Apr 8, 2024
Job added to Upwork: https://www.upwork.com/jobs/~0152270829f1a6f929
melvin-bot commentedon Apr 8, 2024
Triggered auto assignment to Contributor-plus team member for initial proposal review - @mananjadhav (
External)melvin-bot commentedon Apr 8, 2024
Triggered auto assignment to @kadiealexander (
Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.ShridharGoel commentedon Apr 8, 2024
Proposal
Please re-state the problem that we are trying to solve in this issue.
We need to validate if existing items and new changes have the same type before merging in Onyx.
What is the root cause of that problem?
New change.
What changes do you think we should make in order to solve the problem?
Below is the pseudo code which needs to be added in
mergemethod in react-native-onyx:For
set:For
mergeCollection, change the existingKeys logic to below:For
multiSet, this would be the updated method:mananjadhav commentedon Apr 8, 2024
@ShridharGoel Wouldn't we need to check if it's an
Arrayfor the existingValue as well? I am assuming we're using the existing block withOnyxUtils.get(key).then(existingValue.@roryabraham
Logger.logAlertsuffice for the mismatch? do we need any sendActionToDevTools?ShridharGoel commentedon Apr 9, 2024
Proposal
Updated to use similar check for existing value and changes.
dominictb commentedon Apr 9, 2024
Proposal
Please re-state the problem that we are trying to solve in this issue.
We want to:
What is the root cause of that problem?
We don't have such functionality in Onyx yet
What changes do you think we should make in order to solve the problem?
A. Updates for
Onyx.merge:First there're a few considerations:
If existing value is Array, it can be used with Nullish/Array change, but not Object change. If existing value is Object, it can be used with Nullish/Object change, but not Array change.
So just checking
Array.isArrayand compare both is not enough, we need to check the Nullish case as well.We can define a method
isUpdateCompatibleWithExistingValueto validate that the update and the existing value are compatible:With the above considerations, here're the pseudo code of the changes we can make in Onyx here to filter out the invalid change, Log an alert, and check the
changetype compatibility properly:Then below that, when we use
mergeQueue[key]like in here, we should change to usevalidChangesinstead.B. Updates for
Onyx.setIn here, we should get the existing value, then check if the new update is compatible with the existing value, if not, early return.
(we can alternatively get the existing value from
cache.getValueso it can be synchronous)C. Updates for
Onyx.mergeCollectiongetCachedCollectionaccepts an optionalcollectionMemberKeysparam, if it's passed in, we'll use it, if not, we'll use our own existingcollectionMemberKeyshereexistingKeysby usinggetCachedCollectionand passexistingKeysascollectionMemberKeysmergedCollection[key]is compatible with the existing value fromexistingKeysabove, if they are not compatible, Log a warning by usingLogger.logAlert, and do not set toobj(so that update will be ignored)What alternative solutions did you explore? (Optional)
For A., an alternative is to check update compatibility right in here before we push the
changesto themergeQueue. We can get the existing value by usingcache.getValueand check the compatibility usingisUpdateCompatibleWithExistingValue, if they are not compatible, log a warning usingLogger.logAlertand early return, so thechangeswill never end up in the merge queue.66 remaining items
roryabraham commentedon Jun 20, 2024
Ok, so this is definitely out in production right now, and I think it was deployed in #42772.
So that means it's been on prod for about a week. Given the unrelated delays, I suggest we pay this out right away and then close this.
mananjadhav commentedon Jun 22, 2024
@kadiealexander this is ready for payout. Can you update the payment summary?
kadiealexander commentedon Jun 24, 2024
Payouts due:
Upwork job is here.
@dominictb please apply on Upwork, I couldn't find your account.
[-][$250] Add some basic runtime type validation in Onyx[/-][+][Ready for payment][$250] Add some basic runtime type validation in Onyx[/+]mananjadhav commentedon Jun 24, 2024
@kadiealexander I don't need the Upwork payout. I have requested on NewDot.
kadiealexander commentedon Jun 24, 2024
Withdrawn, thanks Manan!
dominictb commentedon Jun 24, 2024
@kadiealexander I submitted the application on Upwork, my profile link is https://www.upwork.com/freelancers/~01f70bed1934fd35d5
Thank you
JmillsExpensify commentedon Jun 24, 2024
$250 approved for @mananjadhav