[06:08:51.428] src/tools/ci/cores_backtrace.sh linux /tmp/cores [06:08:52.125] dumping /tmp/cores/postgres-3-30073.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [06:08:53.791] [New LWP 30073] [06:08:53.791] [New LWP 29922] [06:08:53.791] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [06:08:54.554] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [06:08:55.318] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [06:08:56.546] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [06:08:58.901] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [06:08:58.901] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [06:09:00.766] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [06:09:01.976] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [06:09:03.550] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [06:09:04.609] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [06:09:05.373] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [06:09:06.605] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [06:09:08.216] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [06:09:09.618] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [06:09:10.526] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [06:09:12.057] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [06:09:13.444] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [06:09:13.916] [Thread debugging using libthread_db enabled] [06:09:13.916] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [06:09:14.983] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [06:09:16.372] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [06:09:17.431] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [06:09:18.045] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [06:09:18.810] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [06:09:21.084] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [06:09:22.009] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [06:09:22.773] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [06:09:23.683] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [06:09:24.597] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [06:09:26.185] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [06:09:27.247] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [06:09:28.011] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [06:09:29.396] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [06:09:30.014] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [06:09:31.548] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [06:09:32.309] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [06:09:33.530] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [06:09:34.295] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [06:09:35.364] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [06:09:36.281] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [06:09:37.504] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [06:09:38.120] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [06:09:39.031] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [06:09:39.791] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [06:09:40.282] Core was generated by `postgres: cascade: startup waiting for 000000020000000000000004 '. [06:09:40.282] Program terminated with signal SIGQUIT, Quit. [06:09:40.282] #0 0x00007fc43b8eefc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.309] [Current thread is 1 (LWP 30073)] [06:09:40.427] [06:09:40.427] Thread 2 (Thread 0x7fc439285a40 (LWP 29922)): [06:09:40.427] #0 0x00007fc43b9b1a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.427] No symbol table info available. [06:09:40.427] #1 0x00007fc43b99fe5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.427] No symbol table info available. [06:09:40.427] #2 0x00007fc43b9a0497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.427] No symbol table info available. [06:09:40.427] #3 0x00007fc43b99fcdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.427] No symbol table info available. [06:09:40.427] #4 0x00007fc43b8fba69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.427] No symbol table info available. [06:09:40.427] #5 0x00005635c984c493 in RestoreArchivedFile (path=path@entry=0x7fff40e3a8c0 "`\254\343@\377\177", xlogfname=xlogfname@entry=0x7fff40e3ad10 "00000002", '0' , "4", recovername=recovername@entry=0x5635ca35b9dc "RECOVERYXLOG", expectedSize=16777216, cleanupEnabled=) at ../src/backend/access/transam/xlogarchive.c:177 [06:09:40.427] xlogpath = "pg_wal/RECOVERYXLOG\000\065V\000\000\250\024\033\312\065V\000\000\301'5\312\065V\000\000\271'5\312\065V\000\064\230\247\343@\377\177\000\000\265'5\312\000\000\000\000\030Y(9\304\177\000\000\b\000\000\000\000\000\000\000́5\312\065V\000\000\350\247\343@\377\177\000\000\300\201\065\312\065V\000\000\250\024\033\312\065V\000\000\020\255\343@\377\177\000\000\227\271\065\312\065V\000\000\350\247\343@\377\177\000\000\225\271\065\312\065V\000\000\030Y(9\304\177\000\000\004\037\033\312\065V", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [06:09:40.427] xlogRestoreCmd = 0x5635cb359948 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000002", '0' , "4\" \"pg_wal/RECOVERYXLOG\"" [06:09:40.427] lastRestartPointFname = "00000001", '0' , "2\000\037\033\312\065V", '\000' , "\001\000\000fu\000\000\300f\250;\304\177\000\000`g\250;\304\177\000\000fu\000\000\002\000\000\000\000\000\000\000\060\000\000\000\020\275\217;\304\177\000\000\260\240\343@\377\177", '\000' , "W\261\244;\304\177\000\000O\261\244;\304\177\000\000H\231\065\313\065V", '\000' , "@", '\000' ... [06:09:40.427] rc = [06:09:40.427] stat_buf = {st_dev = 94789024016012, st_ino = 335544320, st_nlink = 140480789540192, st_mode = 0, st_uid = 0, st_gid = 999275792, __pad0 = 32708, st_rdev = 3162674916446634496, st_size = 140733193388032, st_blksize = 94789042542920, st_blocks = 140734282048176, st_atim = {tv_sec = 140734282047760, tv_nsec = 140734282047888}, st_mtim = {tv_sec = 0, tv_nsec = 94789042542920}, st_ctim = {tv_sec = 140480790267031, tv_nsec = 0}, __glibc_reserved = {94789024027816, 140734282050848, 94789025759513}} [06:09:40.427] restartSegNo = [06:09:40.427] restartRedoPtr = 33554472 [06:09:40.427] restartTli = 1 [06:09:40.427] __func__ = "RestoreArchivedFile" [06:09:40.427] #6 0x00005635c985b123 in XLogFileRead (segno=140734282050752, segno@entry=4, emode=emode@entry=13, tli=tli@entry=2, source=source@entry=XLOG_FROM_ARCHIVE, notfoundOk=notfoundOk@entry=true) at ../src/backend/access/transam/xlogrecovery.c:4209 [06:09:40.427] xlogfname = "00000002", '0' , "4\000\000\000\t", '\000' , "\065\000\000\000\000\000\000\000\220\261\343@\377\177\000\000#\246#=\304\177\000" [06:09:40.427] activitymsg = "waiting for 00000002", '0' , "4\000\177\000\000\250\024\033\312\065V\000\000\020\023A\312\065V\000\000\r\023A\312\065V\000\000(\260\343@\377\177\000\000\v\023A\312\065V\000" [06:09:40.427] path = "`\254\343@\377\177\000\000`\254\343@\000\000\000\000\001\000\000\000\000\000\000\000d\023\033\312\065V\000\000pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000|\031\033\312\000\000\000\063\240\253\343@\377\177\000\000\b\000\000\000\000\000\000\000́5\312\065V\000\000\b\000\000\000\000\000\000\000\271'5\312\065V\000\000H\254\343@\377\177\000\000\265'5\312\065V\000\000\250\024\033\312\065V\000\000\301'5\312\065V\000\000\271'5\312\065V\000\000H\254\343@\377\177\000\000\265'5\312\065V\000\000\030Y(9\304\177\000\000F\036\033\312\065V", '\000' ... [06:09:40.427] fd = [06:09:40.427] __func__ = "XLogFileRead" [06:09:40.427] #7 0x00005635c985b313 in XLogFileReadAnyTLI (segno=4, emode=emode@entry=13, source=XLOG_FROM_ANY) at ../src/backend/access/transam/xlogrecovery.c:4337 [06:09:40.427] hent = [06:09:40.427] tli = 2 [06:09:40.427] cell__state = {l = , i = } [06:09:40.427] path = "\220\261\343@\377\177\000\000@\261\343@\377\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "B\212\325\312\065V\000\000\300\212\325\312\065V\000\000\024\257\343@\377\177\000\000\070\022A\312\065V\000\000\240\037\000\000\377\377\002\000\300\212\325\312\065V\000\000\064\257\343@\377\177\000\000G\022A\312\065V\000\000\300\212\325\312\065V\000\000\002\000\000\000\000\000\000\000\245.\033\312\065V\000\000\066\257\343@\377\177\000\000\064\257\343@\377\177", '\000' , "\030\000\000\000\060\000\000\000\020\257\343@\377\177\000\000~/\033\312\065V\000\000\000\000\000\000\000\000\000\000"... [06:09:40.427] cell = [06:09:40.427] fd = [06:09:40.427] tles = 0x5635cb35a078 [06:09:40.427] __func__ = "XLogFileReadAnyTLI" [06:09:40.427] #8 0x00005635c985c50f in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211528, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211504, replayTLI=2, replayLSN=67211504, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3770 [06:09:40.427] oldSource = [06:09:40.427] startWalReceiver = false [06:09:40.427] last_fail_time = 767599553429267 [06:09:40.427] now = [06:09:40.427] streaming_reply_sent = true [06:09:40.427] __func__ = "WaitForWALToBecomeAvailable" [06:09:40.427] #9 0x00005635c985d044 in XLogPageRead (xlogreader=0x5635cb306f20, targetPagePtr=67207168, reqLen=4360, targetRecPtr=67211504, readBuf=0x5635cb356118 "\025\321\005") at ../src/backend/access/transam/xlogrecovery.c:3351 [06:09:40.427] private = 0x5635cb306f00 [06:09:40.427] emode = 15 [06:09:40.427] targetPageOff = 98304 [06:09:40.427] targetSegNo = 4 [06:09:40.427] r = [06:09:40.427] __func__ = "XLogPageRead" [06:09:40.427] #10 0x00005635c986549f in ReadPageInternal (state=state@entry=0x5635cb306f20, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4360) at ../src/backend/access/transam/xlogreader.c:1060 [06:09:40.427] readLen = [06:09:40.427] targetPageOff = 98304 [06:09:40.427] targetSegNo = 4 [06:09:40.427] hdr = [06:09:40.427] #11 0x00005635c98670fa in XLogDecodeNextRecord (state=state@entry=0x5635cb306f20, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [06:09:40.427] RecPtr = 67211504 [06:09:40.427] record = [06:09:40.427] targetPagePtr = 67207168 [06:09:40.427] randAccess = false [06:09:40.427] len = [06:09:40.427] total_len = [06:09:40.427] targetRecOff = 4336 [06:09:40.427] pageHeaderSize = [06:09:40.427] assembled = false [06:09:40.427] gotheader = [06:09:40.427] readOff = [06:09:40.427] decoded = 0x0 [06:09:40.427] errormsg = 0x5635c9859e57 "\351$\375\377\377H\213uhH\205\366t,@\366\306\au&H\215VHH\203\376\270s(H\213FHH\211\342\017\266~8H\211\306\350\254\v\370\377\213T$P\351+\375\377\377H\215=\210\255", [06:09:40.427] #12 0x00005635c9867c9c in XLogReadAhead (state=0x5635cb306f20, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [06:09:40.427] result = [06:09:40.427] #13 0x00005635c98525de in XLogPrefetcherNextBlock (pgsr_private=94789042208080, lsn=0x5635cb36bc48) at ../src/backend/access/transam/xlogprefetcher.c:492 [06:09:40.427] nonblocking = [06:09:40.427] record = [06:09:40.427] prefetcher = 0x5635cb307d50 [06:09:40.427] reader = 0x5635cb306f20 [06:09:40.427] replaying_lsn = 67211096 [06:09:40.427] __func__ = "XLogPrefetcherNextBlock" [06:09:40.427] #14 0x00005635c98569a7 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [06:09:40.427] No locals. [06:09:40.427] #15 lrq_complete_lsn (lsn=67211504, lrq=0x5635cb36b9c8) at ../src/backend/access/transam/xlogprefetcher.c:290 [06:09:40.427] No locals. [06:09:40.427] #16 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x5635cb307d50, errmsg=errmsg@entry=0x7fff40e3f4a8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [06:09:40.446] record = [06:09:40.446] replayed_up_to = 67211504 [06:09:40.446] #17 0x00005635c985d871 in ReadRecord (xlogprefetcher=0x5635cb307d50, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3151 [06:09:40.446] errormsg = 0x5635c9d5e314 "AWAVAUATUSH\201\354X\001" [06:09:40.446] record = [06:09:40.446] xlogreader = 0x5635cb306f20 [06:09:40.446] private = [06:09:40.446] __func__ = "ReadRecord" [06:09:40.446] #18 0x00005635c9860a08 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [06:09:40.446] xtime = [06:09:40.446] ru0 = {tv = {tv_sec = 1714284352, tv_usec = 955937}, ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 1993}, {ru_maxrss = 16436, __ru_maxrss_word = 16436}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 413, __ru_minflt_word = 413}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 16, __ru_nvcsw_word = 16}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [06:09:40.446] record = 0x7fc438b58e50 [06:09:40.446] reachedRecoveryTarget = false [06:09:40.446] replayTLI = 2 [06:09:40.446] __func__ = "PerformWalRecovery" [06:09:40.446] #19 0x00005635c984a42b in StartupXLOG () at ../src/backend/access/transam/xlog.c:5806 [06:09:40.446] Insert = [06:09:40.446] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 738}, nextOid = 14202, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1714284351, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 738} [06:09:40.446] wasShutdown = false [06:09:40.446] didCrash = [06:09:40.446] haveTblspcMap = false [06:09:40.446] haveBackupLabel = true [06:09:40.446] EndOfLog = [06:09:40.446] EndOfLogTLI = [06:09:40.446] newTLI = [06:09:40.446] performedWalRecovery = [06:09:40.446] endOfRecoveryInfo = [06:09:40.446] abortedRecPtr = [06:09:40.446] missingContrecPtr = [06:09:40.446] oldestActiveXID = [06:09:40.446] promoted = false [06:09:40.446] __func__ = "StartupXLOG" [06:09:40.446] #20 0x00005635c9d1004d in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [06:09:40.446] No locals. [06:09:40.446] #21 0x00005635c9d07669 in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [06:09:40.446] pid = [06:09:40.446] #22 0x00005635c9d096e2 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [06:09:40.446] pid = [06:09:40.446] __func__ = "StartChildProcess" [06:09:40.446] #23 0x00005635c9d0f7ac in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x5635cb305de0) at ../src/backend/postmaster/postmaster.c:1364 [06:09:40.446] opt = [06:09:40.446] status = [06:09:40.446] userDoption = [06:09:40.446] listen_addr_saved = false [06:09:40.446] output_config_variable = [06:09:40.446] __func__ = "PostmasterMain" [06:09:40.446] #24 0x00005635c9b3010d in main (argc=4, argv=0x5635cb305de0) at ../src/backend/main/main.c:197 [06:09:40.446] do_check_root = [06:09:40.446] [06:09:40.446] Thread 1 (LWP 30073): [06:09:40.446] #0 0x00007fc43b8eefc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.446] No symbol table info available. [06:09:40.446] #1 0x00007fc43b9a013a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.446] No symbol table info available. [06:09:40.446] #2 0x00007fc43b9b1a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [06:09:40.446] No symbol table info available.