You can set origin-pull rules for buckets in the COS console. When the object you request does not exist in the bucket or a specific request needs to be redirected, you can use the origin-pull rules to access the corresponding data from COS. Setting the origin-pull is mainly used for scenes such as hot migration of data and redirection of specific requests. You can set it according to your actual needs.
Note:
The success rate of pulling data through origin-pull depends on the network environment. Give priority to using IP ranges such as China Telecom, China Mobile, China Unicom, and so on.
Buckets in the Financial Cloud region do not support origin-pull settings.
Synchronous origin-pull is supported only in some public cloud regions, including: Beijing, Chengdu, Guangzhou, Nanjing, Shanghai, Hong Kong (China), Singapore, Jakarta, Seoul, Bangkok, Tokyo, Riyadh, Silicon Valley (US West), Virginia (US East), Sao Paulo, Frankfurt.
Origin-Pull Rules
Triggering Condition
In asynchronous and synchronous origin-pull modes, origin-pull is triggered only when a GET request returns 404. Origin-pull is not triggered for HEAD or other requests.
In redirection mode, users can customize HTTP status codes from 400 to 599 to trigger origin-pull.
Accessing Origin Server
In asynchronous and synchronous origin-pull modes, you can set whether to pass through the QueryString and Header information of accessing COS to the origin server and whether to carry additional Header information when adding a request to the origin server. The redirection mode only supports setting whether the QueryString is passed through.
If the GET range is specified in the GET operation, COS will send an asynchronous request without range in addition to the original request to obtain the complete object data and store it in COS.
Response and Storage
Support the origin server in returning data in chunked encode.
If the origin server returns a 404 status code, it will be passed through to COS and returned to the user. If 3XX Following Policy is enabled when the origin server returns a 3XX status code, it will pull data from another source server. If the origin server returns a status code other than 2XX, COS will return 424.
The files returned from the origin-pull are stored in COS with the same file name as used when requesting the origin server. For example, if the file example.jpg requested by a user is not in the storage bucket, COS will trigger the origin-pull mechanism to the origin-pull address http://origin.com/example.jpg set by the user to pull the file and name it example.jpg in the bucket.
New objects stored in COS contain the following metadata, with the data content following the origin server values:
2. In the left sidebar, click Bucket List to go to the bucket list page.
3. Click the name of the bucket for which you want to set origin-pull to go to the bucket details page.
4. In the left sidebar, choose Basic Configuration > Origin-pull Settings, and click Add Origin-pull Rule.
5. In the pop-up window, configure the following information and click Next:
Origin-pull mode: You can choose the origin-pull mode according to your actual needs.
Asynchronous origin-pull: When a request is made to find a file in COS, and it does not exist, COS will search for the file from the specified origin server. In asynchronous origin-pull mode, after COS finds that the file does not exist, it returns a 302 to the client, which then redirects to the specified origin server, and the file is asynchronously uploaded to the COS bucket.
Note:
In asynchronous origin-pull, the file is not returned directly. Instead, a 302 is first returned to the client, and then the file is asynchronously uploaded to COS.
It is recommended that you enable Follow 302 on the client to pull data from the origin server.
File upload time is affected by multiple factors, and no SLA can be committed. Users who require timeliness are advised to choose synchronous origin-pull.
Synchronous origin-pull: When a request is made to find a file in COS, and it does not exist, COS will search for the file from the specified origin server, return the file to the user, and upload it to the bucket.
Attention:
When synchronous origin-pull is used, timeliness can only be guaranteed when the file is fully downloaded and a status code of 200 is returned. If the client accesses the file through a range request, timeliness cannot be guaranteed.
Redirection: When a specified error is reported when requesting access to a bucket, COS will return the redirection address to the user and will not save the origin server's files. The user side requests access to resources from the origin server through the redirection address.
Origin-pull conditions: You can select the conditions to trigger the origin-pull as required. Triggering the origin-pull requires all configured origin-pull conditions to be met at the same time.
HTTP Status Code 404: When you select asynchronous or synchronous origin-pull, origin-pull is triggered when a GET request returns an HTTP status code of 404. This item is mandatory and cannot be deselected. When you select the redirection origin-pull mode, you can enter an HTTP status code ranging from 400 to 599.
Filename Prefix: The origin-pull rule can be triggered when the requested filename prefix matches. For example, if you set the filename prefix to "prefix", the origin-pull rule will be triggered when a GET request is made to https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/prefix123.jpg and an HTTP status code of 404 is returned. A single origin-pull rule does not support setting multiple filename prefixes.
Filename Suffix: The origin-pull rule can be triggered when the requested filename suffix matches. For example, if you set the filename suffix to flv, the origin-pull rule will be triggered when the URL https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/feedvideo/xx.flv is accessed and an HTTP status code of 404 is returned. A single origin-pull rule does not support setting multiple filename suffixes.
Block Origin-pull Domain: This configuration item is only effective for asynchronous origin-pull and synchronous origin-pull modes. The redirection mode does not support this configuration. When the requested domain name matches, the request will not trigger the origin-pull rule. For example, if you set the blocked domain to examplebucket-1250000000.cos.ap-beijing.myqcloud.com, accessing a non-existent object through this domain will not trigger origin-pull and will directly return 404. You can block up to 10 origin-pull domains. Both the bucket's default domain and custom domains are supported.
Origin-pull protocol: The HTTP protocol used by COS to access the origin server you specify. The options are Force HTTPS, Force HTTP, and Follow Request Protocol.
If you select Force HTTPS/HTTP, COS will access your origin server using the HTTPS/HTTP protocol.
If you select Follow Request Protocol, COS will access your origin server using the protocol you requested COS to use.
Request Parameter: You can choose whether to pass through the queryString request parameter carried during the COS access to the origin server.
Request Headers: You can choose to pass through all request headers or pass through specified request headers. If you choose to pass through all request headers, it is recommended that you configure to block the host header from being passed through. This feature supports one-click configuration.
Pass Through Specified Request Headers: When you select Pass Through Specified Request Headers, you need to configure this item. Here you can add header information in the request that you want to be passed through to the origin server. If you select the redirection origin-pull type, do not configure this item. A single origin-pull rule does not support setting multiple filename prefixes.
Do Not Pass Through Specified Request Headers: You can add header information in the request that you do not want to be passed through to the origin server. If you select the redirection origin-pull type, do not set this item.
Add request headers: You can add additional request headers when origin-pull to the origin server. If you select the redirection origin-pull type, do not set this item.
6. Based on the selected origin-pull mode, configure the following information and click Next:
Asynchronous Origin-Pull
Synchronous Origin-pull
Redirection
Origin-Pull Address: You only need to fill in a domain or an IP address. You can add a port number after a domain name or an IP address. Do not add prefixes such as http:// or https://.
The correct example address is as follows:
abc.example.com
abc.example.com:8080
202.96.128.86
202.96.128.86:8080
Backup Origin-Pull Address: You can set a backup origin-pull address. For the address filling format, see the description of the Origin-Pull Address configuration item. This configuration depends on the bucket region. If the region supports it, it can be configured; otherwise, it cannot.
Address Settings: You can choose whether to set specific files or file paths for redirection. The configuration items are described as follows:
Fixed File Settings: When the origin-pull rule is triggered, all requests will be redirected to a fixed file.
Prefix and Suffix Settings: When the origin-pull rule is triggered, requests will be redirected to the file path with a specified prefix and suffix.
For example, if you specify the prefix as test, when the URL https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/path/prefix123.jpg is accessed, the origin-pull rule is triggered and the request will be redirected to <Origin-Pull Address>/test/path/prefix123.jpg.
For example, if you specify the suffix as .jpg, when the URL https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/path/prefix123 is accessed, the origin-pull rule is triggered and the request will be redirected to <Origin-Pull Address>/path/prefix123.jpg.
Attention:
The configurations of specifying a prefix and specifying a suffix can take effect at the same time.
3XX Following Policy: After you enable it, when the origin server returns a 3XX redirect status code, COS's default processing policy is to follow the 3XX and pull data from another origin server. If you select Disable, resources will not be pulled.
Attention:
When the origin server returns a 3XX redirect to a private network address, COS does not follow it. In synchronous origin-pull mode, COS does not follow the redirect to the private network address. In asynchronous origin-pull and redirect modes, after the private network address is returned, the business side needs to decide whether to follow or disable the redirect.
Origin-Pull Address: You only need to fill in a domain or an IP address. You can add a port number after a domain name or an IP address. Do not add prefixes such as http:// or https://.
The correct example address is as follows:
abc.example.com
abc.example.com:8080
202.96.128.86
202.96.128.86:8080
Backup Origin-Pull Address: You can set a backup origin-pull address. For the address filling format, see the description of the Origin-Pull Address configuration item. This configuration depends on the bucket region. If the region supports it, it can be configured; otherwise, it cannot.
Address Settings: You can choose whether to set specific files or file paths for redirection. The configuration items are described as follows:
Fixed File Settings: When the origin-pull rule is triggered, all requests will be redirected to a fixed file.
Prefix and Suffix Settings: When the origin-pull rule is triggered, requests will be redirected to the file path with a specified prefix and suffix.
For example, if you specify the prefix as test, when https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/path/prefix123.jpg is accessed, the origin-pull rule is triggered and the request will be redirected to <Origin-Pull Address>/test/path/prefix123.jpg.
For example, if you specify the suffix as .jpg, when https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/path/prefix123 is accessed, the origin-pull rule is triggered and the request will be redirected to <Origin-Pull Address>/path/prefix123.jpg.
Attention:
The configurations of specifying a prefix and specifying a suffix can take effect at the same time.
3XX Following Policy: After you enable it, when the origin server returns a 3XX redirect status code, COS's default processing policy is to follow the 3XX and pull data from another origin server. If you select Disable, resources will not be pulled.
Origin Server Response: After you enable this feature, the origin server response, including information such as the status code, will be directly returned. This configuration depends on the bucket region. If the region supports it, it can be configured; otherwise, it cannot.
Origin-Pull Address: You only need to fill in a domain or an IP address. You can add a port number after a domain name or an IP address. Do not add prefixes such as http:// or https://.
The correct example address is as follows:
abc.example.com
abc.example.com:8080
202.96.128.86
202.96.128.86:8080
Address Settings: You can choose whether to set specific files or file paths for redirection. The configuration items are described as follows:
Fixed File Settings: When the origin-pull rule is triggered, all requests will be redirected to a fixed file.
Prefix and Suffix Settings: When the origin-pull rule is triggered, requests will be redirected to the file path with a specified prefix and suffix.
For example, if you specify the prefix as test, when https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/path/prefix123.jpg is accessed, the origin-pull rule is triggered and the request will be redirected to <Origin-Pull Address>/test/path/prefix123.jpg.
For example, if you specify the suffix as .jpg, when https://examplebucket-1250000000.cos.ap-chengdu.myqcloud.com/path/prefix123 is accessed, the origin-pull rule is triggered and the request will be redirected to <Origin-Pull Address>/path/prefix123.jpg.
Attention:
The configurations of specifying a prefix and specifying a suffix can take effect at the same time.
Redirect Code: 301, 302, and 307 are optional, with 302 being the default.
7. Confirm that the configured origin-pull rule is correct and click OK.
After a rule is added, the system will assign the highest priority to the new rule. COS will perform origin-pull according to the highest-priority rule. Additionally, you can adjust the priority by clicking the Modify button on the rule list page.
Attention:
After a high-priority source fails, it will not revert to a low-priority origin server address. Instead, it will revert to the backup origin server address of the origin server address.
Examples
Background
A user with APPID 1250000000 created a bucket named examplebucket-1250000000 and enabled the CDN acceleration access domain name:
examplebucket-1250000000.file.myqcloud.com
Set the bucket origin-pull address to:
abc.example.com
Store the picture picture.jpg at the origin server http://abc.example.com.
Client's first visit (with synchronous origin-pull disabled):
When COS finds that it cannot hit the object, it returns a 200 HTTP status code to the client and redirects to the following address:
http://abc.example.com/picture.jpg
At this point, the object is provided to the client by the origin server, ensuring access. Meanwhile, COS copies picture.jpg from the origin server and saves it to the root directory of the bucket examplebucket-1250000000.
