By Shift Team in Newsletter on Oct 3, 2018
Today’s is a very important development update. Firstly, we will be revealing significant information concerning the pricing model that will underpin the locking mechanism used to allocate storage on our platform. Following that, and as promised in our previous newsletter, we will be discussing advances that have been made on the task of blockchain integration. Finally, it is with some regret, though far greater optimism, that we shall end this summary of our progress by explaining a necessary amendment to the release schedule of our decentralized storage and web hosting solution.
1. Pricing Models
The Shift Project is building a decentralized storage platform with which users will be able to grant drive space to the network as storage providers, as well as make use of the storage resources of others in order to decentralize their data and web hosting apparatus. As the provision of drive space requires the deployment and maintenance of software and hardware, which takes time and money, an ecosystem such as this will only function in a healthy and sustainable manner if there is a financial system in place that rewards participants for granting the services underpinning it. Now, thanks to the emergence of the world’s first decentralized token, Bitcoin, and the blockchain industry that has grown out of it, we, at Shift, have developed a trustless exchange platform on which our token can be used to purchase decentralized storage.
In the process of working on our decentralized storage solution, we have carefully considered the potential payment methods incorporated into its financial system, methods that allow our platform’s storage users to engage in an equitable exchange of resources with those providing storage infrastructure. A result of these deliberations are two principal models that would enable the use of SHIFT tokens as a payment medium for staking a claim to data capacity. These two models can be succinctly summarized as follows:
Direct Transfer Model
In the first model we considered, storage users would directly pay storage providers through a transfer of SHIFT between accounts held in their mutual possession. This model requires a ‘decentralized storage marketplace’ where storage users and providers are able to post requests and offers, and subsequently respond to these by transacting SHIFT in order to take them up.
In the case of the second model, storage users are responsible for purchasing SHIFT and using an integrated lock function to stake a claim to a corresponding number of available bytes of space drawn from the storage cluster’s pool. Here, the value of SHIFT in bytes is determined by a predetermined economic formula. This system frees storage providers from the need to repost offers on a marketplace once they have become a participant in the network, because their storage capacity will be added to the general supply made available to storage users. The system would be designed so that storage providers are rewarded with a proportion of tokens generated concurrently with those produced by each forging delegate. The number of newly released tokens granted as part of this compensation system would need to also be determined by a dynamic economic formula.
Now that we have described the two payment models in brief, let’s analyze them while paying particular attention to two important properties that play a role in determining whether a pricing model is capable of supporting adoption: (1) fair pricing and (2) user friendliness.
When considering the first indicator of viability for adoption, a prominent assumption is that a healthy data storage platform should be priced in such a way as to mutually benefit both storage providers and storage users. We believe that this can be best achieved by ensuring that our platform sees to it that supply and demand are consistently kept at an optimal balance.
Open marketplaces provide a forum for competitive trading on the part of their participants. If buyers are willing to pay X, then a proportion of sellers will generally appear who are willing to accept X as long as the transaction remains a net gain (and vice versa). Competition brings about a balance between the prices people are willing to pay and those prices people are willing to sell at, as long as the conditions of the market remain relatively stable and supply and demand do not fluctuate in a too volatile manner. This is achieved because the actions of a plurality of market participants, some more willing to accept a less favorable exchange, will ordinarily temper aspirations for maximizing gain to a point where a halt in trading occurs. Thus, marketplaces, possessing a dynamic quality, may contribute to bringing about fair pricing.
Conversely, the difficulty in establishing an equitable formula-driven model is that, due to it removing competition between parties on either side of the transaction, it must be designed in such a way as to automate the process of meeting the expectations of both buyers and sellers. If one side of the exchange feels that the price is unfair, they will be disinclined to use the platform until the price is somehow corrected. This means that the model has to be constructed in such a way as to allow rapid self-correction, while also being robust in the face of potential price manipulation. The process of designing and implementing a formula-driven price model is therefore a very complex one.
So, in summary, a decentralized marketplace implies a heuristic method of achieving a ‘fair’ price, thus requiring greater user participation, while a formula-driven model has the advantage of automation, rendering it ‘fair’ through the impartiality that results from it being based on advanced economic formulas.
In order to conform with the second indicator, user friendliness, we will seek to ensure that our services are barrier-free for both parties. This will be achieved through the release of software that appeals to a broad array of potential customers in its simplicity, employing intuitive interfaces that encourage sustained engagement with that platform’s services.
We are aware that even cryptocurrency enthusiasts, who have often spent time trading assets on marketplaces, may be put off by the prospect of engaging in the ‘trading’ necessary to procure storage space using a marketplace model. The large number of continuously changing buy and sell orders, different methods of displaying past trades and market trends, etc. could all add up to a chore that inhibits people from placing even their first order.
The benefit of a formula-driven model is that all a storage user needs to do in order to conduct a transaction, is engage directly with the user interface responsible for allowing the input of a desired quantity of storage space. This makes the process of purchasing data capacity a simple one, that is not dependent on another individual fulfilling ‘their side of the bargain’. As for the storage provider, all that the formula-driven model requires that they do is designate a Shift wallet to be credited when the automated distribution of compensation routinely occurs, in addition to sending a transaction that confirms the assignment of the data capacity of their storage device to the pool. We believe that this minimalist approach will incentivize both parties to use the Shift platform, contributing to its rapid adoption by storage users and storage providers alike.
So, in summary, a decentralized marketplace requires that a platform incorporates complex functionality to allow mediation between buyers and sellers of storage capacity, while a formula-driven model grants greater ease-of-use by mediating between those wishing to offer storage capacity and those who would like to make use of it.
Both a decentralized storage marketplace and a formula-driven pricing model present challenges in their implementation. A decentralized storage market may have a straightforward exchange mechanism and greater ‘choice’, but it adds many more layers of interaction that all need to be developed, built and managed. What is more, maintaining such a blockchain infrastructure would require a core capable of rapidly handling thousands of transactions. One need only look at the lack of fully operational decentralized exchanges that have been built based on blockchain technology. On the other hand, a pricing model based on a price formula may be simpler to implement, core-wise, but requires a great deal of testing to ensure that it meets the expectations of both buyers and sellers, who are no longer interacting directly.
We, at Shift, presently believe that the benefits of the formula-driven pricing model are worth exploring for its potential to contribute to the rapid adoption of our services and growth in our network. Automating the process of price determination through mathematics and thus removing the necessity that storage users and providers need to actively participate in a marketplace, may be an excellent means of removing the barrier to entry that might otherwise exist were newcomers to be faced with an elaborate trading platform. For these reasons, we have created the first and most vital aspects of the formula-driven pricing model system: a token lock function for storage users to claim capacity and the mathematical formula necessary to actuate it by self-regulating the cost in SHIFT required to carry out this function.
Once these two components of Shift’s network technology and financial system are successfully tested and deployed, the complementary task of finalizing the means by which storage providers will be rewarded for their services will begin. That task will entail implementing a control mechanism for storage providers to verify that they’re correctly hosting the content the network has assigned them, as well as enabling a dynamic block reward model that will distribute a quantity of each block reward to the storage providers that successfully and consistently pass the verification process. This second component of the financial system will be the cornerstone of our next development milestone.Please note: While we intend to implement the formula-driven pricing model first, we have not excluded the possibility that we may, at a later date, re-architecture our platform to render it more appropriate for the implementation of a decentralized marketplace.
Shift’s Locking Mechanism
Now that we have discussed the options available in terms of methods for determining the price of storage resources on our network, as well as explained our reasoning behind initially focusing our efforts on a formula-driven pricing model, we would now like to explain the mechanics behind the LOCK transaction. This is where the SHIFT token acquires its primary function, as it will serve as a medium for staking a claim to available storage space within our IPFS swarm. Merely possessing SHIFT does not entitle one to storage. Rather, tokens must be locked using a LOCK transaction that, in making those tokens unavailable for other uses, functions as a request for a proportion of the Shift pool’s storage capacity. This drive space will only remain available to the storage user for as long as their tokens remain locked, with an UNLOCK transaction being used to free the tokens and return the drive space to the platform’s available pool. Thus, with our upcoming core (client) and wallet update, there will be a fundamental difference between an account’s available token balance and its locked token balance.
With the imminent release of our ‘LOCK transaction-enabled Shift core’ onto the testnet, we would like to provide a detailed explanation of one of the most significant additions we have made to our platform, the mathematical formula used within our dynamic formula-driven pricing model:
P = C * ( X / Y ) * Z * R
- P = [SHIFT] = Amount of tokens to lock for Z amount of drive space.
- C = [dimensionless] = Constant.
- X = [byte] = Total amount of the platform’s drive space that is claimed (demand).
- Y = [byte] = Total amount of the platform’s drive space that is on offer (supply).
- Z = [byte] = Amount of drive space requested by user.
- R = [SHIFT/byte] = Pre-initiated token/byte ratio (conversion factor).
Please note: This price formula only concerns the means by which the quantity of SHIFT locked by the storage user is calculated, and is not used to establish the compensation paid to storage providers for rendering their services. The establishment of the mechanisms for this will be a part of the next development milestone.
Furthermore, this is the frontend side of the formula. The backend only receives a LOCK transaction with an amount of SHIFT and takes account of the transaction fee. The frontend outputs the value of P and that value is used by the backend to calculate Z. The backend side of the formula becomes:
Z = P / ( C * ( X / Y ) * R )
- P, or Price, is principally based on the ratio between demand X and supply Y. If demand for storage (numerator) increases, the X/Y fraction increases, causing the value of P to increase. If supply of storage (denominator) increases, the X/Y fraction decreases, and thus the value of P will also decrease.
- Z is the amount of drive space that is requested by the storage user. This is the only variable in the frontend side of the formula that has to be assigned by the user themself.This means that the user interface for the LOCK transaction will only have one input field, the preferred (Z) amount of drive space that the storage user wishes to claim (or its corresponding value in SHIFT). The user interface will be similar to those commonly encountered on cryptocurrency exchanges when conducting a market order, where the user enters the amount of tokens they want to buy and the corresponding cost in the transacting currency is displayed automatically. The complexity of the formula for P is thus made easily comprehensible.Please note: In the user interface it will be possible to enter the P amount of tokens too, and the corresponding Z amount of drive space will be automatically displayed. However, we think it is convenient to enter the amount of drive space, as that is what the user is procuring.
- Constant C, being a multiplying factor, determines the magnitude of the effect of the X/Y ratio. For now, C will be a constant variable, although, in the future, it could be altered into a dynamic variable based on several parameters, such as PIN activity, number of storage nodes and accounts with a locked balance.It is likely that when the formula is put into action on the testnet, the constant C will be set to 1/3 or 0.3 (rounded). This is due to the replication factor of our IPFS network, that is, by default, set to three and therefore requires each file to be stored at three different locations at all times. The consequence of this is that with every pin request, three times as much drive space is used when compared to the actual asset size of that pin. In other words, a storage user can only use 1/3 of their allocated storage for unique data. In order to ensure that the formula correctly compensates for this, the figure expressed in P is reduced to one third of the X/Y ratio.
- The conversion factor R is required to ensure that the value of P has the correct units for being appropriately expressed according to the numeric properties of the SHIFT token, and can thus be considered to be the pre-initiated SHIFT/byte ratio. It is likely that R will be set to 1 SHIFT per 1 Megabyte (1 MB equals 1,000,000 bytes).Please note: This conversion factor concerns the SHIFT testnet token and does not necessarily imply that the value of R will be set to the same figure when we launch the formula-driven pricing model on the mainnet. For the testnet launch, we have made the decision to set R to 1 SHIFT/MB in order to prevent testnet users from claiming excessive amounts of drive space. This is a necessity because storage providers that grant drive space to the testnet platform do so without being financially compensated, placing constraints on the supply available within the testing ecosystem. Therefore, the distribution of testnet tokens is principally controlled by the Shift Team, with it being our intention to distribute a proportion of the tokens to long term and active community members, but with the majority granted to special partners that will be using our decentralized storage suite.
- Demand X, the total amount of the platform’s drive space that is claimed, will be calculated by the function SELECT SUM (locked_bytes) FROM mem_accounts, where locked_bytes represents the total confirmed locked bytes held against an account. To put this less technically, this function sums the number of locked bytes of every account that has had its allotment of claimed bytes confirmed by the blockchain.
- Supply Y is the total amount of offered drive space available on the IPFS swarm. The Y value will be calculated based on the two swarm statistics calls (API) TotalStorage and UsedStorage, by subtracting the two from each other: TotalStorage – UsedStorage = FreeStorage, which is the figure for Y. FreeStorage will be calculated by the swarm peers and thus will itself be a swarm statistics call. TotalStorage is the SUM of the drive space that each swarm peer has made available for our IPFS network. UsedStorage is the SUM of the space that each swarm peer is using to store content on our IPFS network.Some might wonder why we haven’t defined Y, the supply, as TotalStorage, which would also have been an option. However, it is our opinion that this would be an inappropriate definition for Y because it would not take into account the fact that a proportion of the storage supply may already be in use. To elaborate on this potentially contentious definition, during an instance of scarcity of storage capacity on the swarm, demand X and supply Y will be almost equal to each other and thus the X/Y ratio will approach 1. As a result, the SHIFT/byte ratio will be close to the pre-initiated value, R. This is an undesirable occurrence because scarcity would ideally mean that the SHIFT/byte ratio increases relative to R. In other words, the user should be required to lock more tokens for the same amount of drive space when there is a diminished supply. This need to lock more tokens should, in theory, stimulate the value of the token to increase, in return encouraging storage providers to grow in number or for the quantity of drive space they offer to the network to increase. This ideal outcome can only be achieved if Y is defined as FreeStorage and not as TotalStorage. Hence, it is our preference to calculate FreeStorage instead and define that as Y.
The dependence of price P on the ‘swarm statistics-ascertained Y’ is one of the concrete examples of blockchain integration with Shift’s storage network. LOCK transactions will, like any other transaction type, be processed and verified by delegate nodes, but this specific transaction relies on real time data from the swarm peers.
At first glance, this construction might seem problematic as delegate nodes would not normally be able to rebuild the blockchain as a swarm peer would typically respond with the current FreeStorage and not with historic values. This is because the cluster has been built to be stateless, meaning it doesn’t have a database containing the necessary historic values that a delegate node would need to verify Y during rebuilding. As a result, even regular transaction and block validation by forgers could become troublesome owing to different forgers asking different swarm peers for their statistics, and these swarm peers stating differing values of Y. Events that might cause this to occur include a swarm peer having trouble connecting to its peers, a timeout, or the values for UsedStorage (and thus for FreeStorage) changing rapidly due to volatility in PIN and UNPIN requests.
To remedy this problem, we had to come up with a clever solution that allows forgers who are responsible for validating LOCK transactions to always maintain consensus about the value of Y. What we intend to do is introduce a mechanism that will average swarm statistics over a delegate round (45 minutes and 27 seconds). For any LOCK transaction sent during a delegate round, the value of Y that will be used to process the transaction is the mean average of all 101 Y values (minus outliers) that were used for forging a block during the previous delegate round. Below is a description of the workflow that will be followed in order to calculate Y during the span of one delegate round.
- A new column in the peer table of delegate nodes will be added, a storage boolean (a binary variable, having two possible values called ‘true’ and ‘false’). As a result, delegate nodes will have a few swarm peers listed in their config in order to find additional swarm peers.
- Every 5 minutes, each forger will make an API call to a random swarm peer in order to request the FreeStorage of the entire IPFS swarm.
- The swarm peers will sum on the fly, an alternative way of calculating to ‘polling’ or ‘caching’, which means they calculate the FreeStorage value on demand, only by request of the forgers, instead of storing the value in a database. So, with summing, the calculation is generated on demand and implies that each swarm peer asks every other peer how much drive space it has set as available within its config (TotalStorage) and how much drive space it is using (UsedStorage). Each peer then sums up all the values from all other peers for both TotalStorage and UsedStorage separately. Following this, each peer will then subtract the latter value from the former in order to calculate Y: TotalStorage – UsedStorage = FreeStorage. The value of Y is the info that is returned to the forger that made the API call.
- Each forger will store this information in cache and keep its cache up to date by making such API calls for FreeStorage every 5 minutes within the delegate round. That means that during each delegate round the cache is updated with 9 values of Y in total.
- During a delegate round, each of the 101 forgers is assigned to forge a specific block. If a forger needs to forge a block, it will use the latest value of Y that it has stored in cache. When producing the block, the value of Y is recorded inside the block. The remaining 100 forgers (witnesses) that verify the block of the block producer have their own values of Y stored in cache. In theory, these should be exactly the same value as is contained in the block forged by the forging delegate. However, in practice, they may very well not be. Therefore, the 100 witnesses will verify the block based on their own value of Y and how ‘confident’ they are that it is within an acceptable range (e.g. 5% of their own value).
- If a forger needs to process a LOCK transaction, it will take the mean average of all 101 Y values that were recorded during the block forging of the last delegate round, but with outliers removed. In other words, the forger adds up all 101 values and then divides this figure by 101 to ascertain the mean average from these values, but with outliers excluded based on an interquartile range or their values having fallen outside of a permitted, standard deviation.
- If a forger needs to rebuild the blockchain, and starts reverifying LOCK transactions, they will base it on the average Y of the previous round. So, if a LOCK transaction within a block doesn’t use the average Y defined in the previous round, then the transaction and block will be rejected. The only blocks that could be faked are the initial ones because they are not yet informed by a ‘history of Y’. We avoid this issue by planting a seed value of Y in the code.
Shift’s Unlocking Mechanism
Now that we have discussed the locking mechanism in detail, let’s go over how it’s complementary function, unlocking tokens, works. Our objective is to grant users the freedom to unlock any proportion of the SHIFT that makes up their locked balance, a figure that will be the total SHIFT locked in the LOCK transactions that they have already sent (minus the SHIFT returned from subsequent UNLOCK transactions). We have therefore decided to make the UNLOCK transaction pro-rata, which means that every single token that makes up a locked balance will, when unlocked, return a proportional number of bytes to the storage pool. Thus, the total amount of tokens locked by the user will be considered a single aggregated value, even if these tokens were locked at different points in time and sent using different LOCK transactions.
The UNLOCK formulas:
bytes_unlocked = (shift_submitted_for_unlock / total_shift_locked) * total_bytes_locked
remaining_shift_locked = total_shift_locked – shift_submitted_for_unlock
shift_returned_to_available_balance = shift_submitted_for_unlock – tx_fee
For example: If you sent 50 SHIFT and locked 100 MB, then another 100 MB for 55 SHIFT, the value of total_shift_locked will be 105 SHIFT and the value of total_bytes_locked will be 200 MB. If you then submit 30 SHIFT for an unlock, you would return ( 30 / 105 ) * 200 = 57.142857 MB (bytes_unlocked). Furthermore, 75 SHIFT will remain locked while the 30 SHIFT (minus a transaction fee) will be returned to your available account balance.
The pro-rata implementation is very simple for the frontend, because the frontend can instantly display how many tokens the user will receive based on how many they wish to unlock. Furthermore, the frontend formula is two-sided. This means that the user can either choose to input the amount of tokens they wish to unlock, or choose to input the amount of bytes they would like returned. For locking, users may lock their tokens for one specific goal only, such as to obtain a required amount of drive space, while for unlocking, users may wish to unlock their tokens to either return a specific amount of drive space that they’re not using or because they want to use a specific amount of tokens for a purpose other than storage.
Using the same example: If a user wanted to return 57.142857 MB, you would obviously also unlock ( 57.142857 / 200 ) * 105 = 30 SHIFT (minus a small transaction fee).Please note: The UNLOCK transaction will ultimately always be based on the amount of tokens sent in the transaction. In the example where a desire for the return of 57.142857 MB (30 SHIFT) motivated the unlock, then what the frontend will actually send to the backend is a request to unlock 30 SHIFT (minus a small transaction fee).
Furthermore, it is important to note that any UNLOCK transaction is restricted based on the current pins the account has in place. For the UNLOCK to be valid, the user has to have enough of his locked storage space unused/unpinned. Using the same example where 200 MB was claimed, the UNLOCK transaction would only be valid if the proportion that is used/pinned is not greater than 200 – 57.142857 = 142.857143 MB.
2. Blockchain Integration
Now that we have gone into some detail on the pricing model, as well as explained the locking mechanism, it’s time to discuss the latest progress that has been made on blockchain integration.
- Rob L. has now completed the required database migrations to add support for tracking swarm statistics and tracking the lock/pin functions. The database code has been written to insert LOCK and UNLOCK, as well as PIN and UNPIN transactions into the memory table (that stores previously requested information), allowing fast lookups of these transactions. Following this, Rob was also able to update the memory accounts table (that stores account information of those who have sent the above transactions) so that it is able to track pending and confirmed lock data. This allows the system to keep a running total of a user’s locked and available balances.
- The transaction code now tracks available balances so that it is not possible to perform transactions other than PIN, UNPIN and UNLOCK with an account’s locked balance.
- The logic classes for LOCK and PIN transactions are now complete, though they have yet to be hooked up to the module classes.
- Locking an available balance credits an account’s locked balance and proportionally increases its quota of total storage bytes. Conversely, unlocking a locked balance will reduce the user’s storage byte allocation based on the ratio of locked SHIFT versus the transaction amount, minus all applicable fees. All requests are paired with SHIFT and bytes, with the value of bytes and SHIFT calculated using the formula of our pricing model.
- The constants.js introduces the formula variables so it can be controlled by block height.
- The PIN, UNPIN, LOCK and UNLOCK endpoints are all defined.
Things Rob intends to complete in the coming week:
- Hook up the logic modules to the endpoints.
- Complete the block validation and round averaging of the LOCK transaction.
Once those two final tasks are complete, we’re good to go and we will enter the testing phase on our private network. It is important to note that, while the individual components have been successfully tested locally, once the constituent parts are pulled together it will be necessary initially to carefully monitor them to ensure they perform as expected as a whole.Please note: PIN and UNPIN transactions will be explained in detail within a subsequent development update.
3. Rescheduled Release of the Decentralized Storage Solution
In our July 17th, 2018, newsletter, we, the Shift Team, announced our expectation that by the conclusion of Q3 of this year, we would have released a functioning decentralized storage solution onto our project testnet. This was a goal we aspired to throughout the subsequent months, a goal we used to challenge and motivate ourselves to achieve new and exciting advances while we worked during this long lull in market vitality. However, there is no denying that there remains some final tasks to be completed before the latest iteration of our storage technology is ready to be moved from our private network, to the public domain of the Shift testnet.
We ask you, our community, to endure, with us, just a while longer, the wait between now and the moment we share our game-changing product with the world. Once we have completed the testing of the code, ensuring that the workflow of the software allows functions to operate correctly and the system to maintain its integrity under ideal circumstances, we will be ready to put it into action in an environment to which the masses will have access.
We, as this newsletter may demonstrate, expend a great deal of energy on contemplating the optimal way of theorizing and actualizing solutions to problems that, until very recently, had not troubled a human mind. What you must be in no doubt of, is that solutions are being found and the final steps in preparing our decentralized storage and hosting solution for public use are being completed. By way of an example, Craig C. has now finished work on his first version of the Shift IPFS Cluster Pin Manager. In fact, the contribution made by this software application to our product’s operation is such that we have decided it warrants a name that effectively conveys the groundbreaking role it plays in ensuring that data remains accessible and uncensorable as long as the uploader desires. For that reason, we would like to introduce Phoenix, the Shift peer-to-peer library and PIN command issuing module, and devote our next newsletter, to be published prior to our revised testnet launch, to explaining the full significance of its role. For Phoenix, along with blockchain integration, is one of the final pieces of the puzzle to be added to our fledgling platform before its launch, before presenting our work to you, the Shift Community.
The Shift Team