postgresql – postgres: could not receive data from WAL stream: ERROR: requested WAL segment has already been removed

Configuration: postgres 9.6 with a 3 cluster node. db1 is the master, db2 and db3 are replicas. WAL files are archived in AWS S3 using custom pgrsync tool. Cluster managed by patroni. The archive_command and restore_command is properly configured on all the nodes.

To simulate: On db1, do heavy writes (like vacuum a large table) and then stop db1 by sudo systemctl stop patroni). db3 becomes the new leader. db2 requests more WAL files, which it gets via the proper restore command from AWS S3, becomes replica to db3.

Now, start db1 again by (sudo systemctl start patroni). But db1 (the old leader and the new to-be-replica) never comes up as a replica and gives the error message could not receive data from WAL stream: ERROR: requested WAL segment 0000002400053C55000000AE has already been removed. This error message is reported by db3 (the leader), which db1 just logs it.

So, let’s see the timeline. Initially db1 was in timeline 35 (0x23) and did write the following files to archive:

0000002300053C55000000AE   (TL is 23 and archive at 53C55/AE)
..
0000002300053C5600000043   (TL is 23 and archive at 53C56/43)

db1 is stopped at this point.
db3’s logs show this:

received promote request
redo done at 53C56/A3FFC8C0
selected new timeline ID: 36
database system is ready to accept connections

and db3 copies the following files to archives

00000024.history
0000002300053C56000000A3.partial
0000002400053C56000000A3
0000002200053C55000000AE

As db3 became leader, db2 starts the process to become replica to db3 (which it successfully becomes) and here is the summary of the logs:

restored log file "00000024.history" from archive
restored log file "0000002300053C55000000AE" from archive
restored log file "00000023.history" from archive
redo starts at 53C55/AE0001A8
restored log file "0000002300053C55000000AF" from archive
...
restored log file "0000002300053C5600000042" from archive
Retrieving 0000002300053C5600000043 to pg_xlog/RECOVERYXLOG FAILED  (log by pgrsync)
...
Retrieving 0000002300053C56000000A2 to pg_xlog/RECOVERYXLOG FAILED
restored log file "0000002400053C56000000A3" from archive
consistent recovery state reached at 53C56/A3FFE900

db1 is started now and here are the logs:

LOG: database system was interrupted while in recovery at log time 2021-01-28 04:08:01 UTC 
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target
LOG: invalid record length at 53C55/AE0001E0: wanted 24, got 0
LOG: started streaming WAL from primary at 53C55/AE000000 on timeline 36
FATAL:  could not receive data from WAL stream: ERROR:  requested WAL segment 0000002400053C55000000AE has already been removed
... and repeats this over and over

Points to note:

  1. 0000002400053C55000000AE was never written to archives by any postgres node. The old leader (db1) copied the archive 0000002300053C55000000AE (note: 0023, not 0024) before it was stopped.
  2. The new leader (db3) copied 0000002200053C55000000AE (note: 0022, not 0024)
  3. max_wal_size is set to 1024 on all nodes.
  4. After db3 became the new leader, there was hardly any activity on the nodes. db3 only writes WAL files every 10 mins (archive_timeout=600s).

Questions:

  1. Is there any thing wrong in the configuration that makes the the old leader asking for a WAL segment, which the new leader does not have?
  2. How to restore the old leader (db1) at this state, without having to erase and start over?

php – Rank Math – how to include the Rich Meta Data on my own page?

I have a WooCommerce store with the Plugin “SEO by Rank Math” installed.

This plugin creates Rich Meta Data and places it on my product pages.

Now, I have created an AMP version of my WooCommerce Store in Core PHP and need to include the Rich Meta Data there as well.

I don’t know how to trigger the function of Rank Math to get the same JSON output on my AMP page.

Who can help?

java – Multiple services with common code that gets a map and sets objects’ prop using the map data

I have the following service class:

@Service
@RequiredArgsConstructor
public class AutoVendorNameService  {

    // client of a 3rd party web service
    private final AutoClient autoClient;   
    
    public void setAutoVendorName(ReportCars cars) {  
        
        // autoVendorNamesById: car id is key and vendor name is value  
        Map<String, String> autoVendorNamesById = 
                autoClient.getAutoVendorName(cars.getIds()); //cars.getIds() returns Set<String>
                
        for (ReportCar car : cars.getReportCars()) {            
            String autoVendorName= autoVendorNamesById.get(car.getId());            
            car.setAutoVendorName(autoVendorName);
        }
    }
    
}

Then, I have other services that have similar functionality. For example,

@Service
@RequiredArgsConstructor
public class AutoDealerNameService  {

    // client of a 3rd party web service interface
    private final AutoClient autoClient;   
    
    public void setAutoDealerName(ReportCars cars) {  
        
        // autoDealerNamesById: car id is key and dealer name is value  
        Map<String, String> autoDealerNamesById = 
                autoClient.getAutoDealerName(cars.getIds()); //cars.getIds() returns Set<String>
                
        for (ReportCar car : cars.getReportCars()) {            
            String autoDealerName= autoDealerNamesById.get(car.getId());            
            car.setAutoDealerName(autoDealerName);
        }
    }
    
}

There are few services that have a different value type in the map but follow the same boilerplate. For example, I have a AutoPriceService that Map<String, BigDecimal> autoPriceById.

Is there a way to write a generic method to help reduce this boilerplate duplicative code? I’m using Java 8.

Half dead screen, trying to recover data (device not showing on file explorer)

i have a somewhat old Samsung s7 edge phone which (strangely enough) has only the lower half of the screen not working. This is, the whole screen lights up and displays eveything, but ONLY the lower half is unresponsive to touching. As such, i cannot unlock it and turn usb debugging on. It is not rooted nor do i know how to do it, i dont really know a thing about coding. I have seen many previous threads and web pages about this, but at most i am very confused and dont want to brick it. It doesnt have any really important files, but very old GBA roms id really like to keep, as well as some photos and pictures i may have. Any help apreciated! (and sorry for my weird english. i am not a native speaker, i dont know if it may have messed up)

data structures – Algorithm chalenge: find minimum range to cover all houses with light

N houses are located on a straight street which has K streetlights. Each streetlight has range R. Given N and K how to find minimum value for R?

N is an array containing location of each house.
K is an integer value.

I am unable to identify the algo for this. I tried solving by taking location of 1st and last house, then dividing by K, this will give me max range of each streetlight. However, assume N=[3,4,5,10,13] and K=3. With division this wont work.

swift – My array is empty when downloading data from Firestore

When trying to receive a list of items back from a users favourites collection my array is empty. I have used this code before and im not sure why its not populating.
Below is the ViewController for the favourites im not sure why it is not displaying the favourites. There is two functions that are on the favourites model class. Hopefully someone can spot what I am missing.

//MARK: Outlets
@IBOutlet weak var favouritesTV: UITableView!

@IBOutlet weak var noUserView: UIView!
//MARK: Vars
var favourites : Favourites!
//var allRecipes : (Recipe) = ()
let db = Firestore.firestore()
var listener: ListenerRegistration!
var allItems: (Recipe) = ()

override func viewDidLoad() {
    super.viewDidLoad()
    favouritesTV.delegate = self
    favouritesTV.dataSource = self
    
    favouritesTV.tableFooterView = UIView()
    if User.currentUser() != nil {
        loadFavouritesFromFirestore()
        print("the favs", allItems)
    }else {
        showLoginView()
    }
}

override func viewDidAppear(_ animated: Bool) {
    super.viewDidAppear(animated)
    //Checkuser loggin
   
}
//MARK: - Download favourites
private func loadFavouritesFromFirestore() {
    
    downloadFav(User.currentId()) { (favourites) in
        
        self.favourites = favourites
        self.getFavouritesItems()
    }
}

private func getFavouritesItems() {
    
    if favourites != nil {
        
        downloadItems(favourites!.recipeIds) { (allItems) in
            
            self.allItems = allItems
            self.favouritesTV.reloadData()
           
        }
    }
}

//MARK: Actions
//Show Login View

private func showLoginView(){
    let loginView = UIStoryboard.init(name: "Main", bundle: nil).instantiateViewController(identifier: "WelcomeVC")
    
    self.present(loginView, animated: true, completion: nil)
    
}

//MARK: Table View Func
func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
    return allItems.count
}

func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
    let cell = favouritesTV.dequeueReusableCell(withIdentifier: "recipeCell", for: indexPath) as! RecipeCell
    cell.configRecipeCell(allItems(indexPath.row))
    
    return cell
}

func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
    showRecipeView(withRecipe: allItems(indexPath.row))
}

func tableView(_ tableView: UITableView, canEditRowAt indexPath: IndexPath) -> Bool {
    return true
}

 func tableView(_ tableView: UITableView, commit editingStyle: UITableViewCell.EditingStyle, forRowAt indexPath: IndexPath) {
     if editingStyle == .delete {
        let itemToDelete = allItems(indexPath.row)
        
        allItems.remove(at: indexPath.row)
        favouritesTV.reloadData()
        
      //  removeItemFromBasket(itemId: itemToDelete.id)
        
        updateFavouritesInFirestore(favourites!, withValues: (kRECIPEIDS : favourites!.recipeIds)) { (error) in
            if error != nil{
                print("error updating favourites", error?.localizedDescription)
            }
            
        }
     }
 }
//MARK: Functions
//Show recipe detail view
private func showRecipeView(withRecipe: Recipe) {
    let recipeDetailVC = UIStoryboard.init(name: "Main", bundle: nil).instantiateViewController(identifier: "recipeDetailView") as! RecipeDetailsViewController
    recipeDetailVC.recipe = withRecipe
    
    self.navigationController?.pushViewController(recipeDetailVC, animated: true)
}


func downloadItems(_ withIds: (String), completion: @escaping (_ itemArray: (Recipe)) ->Void) {
    
    var count = 0
    var itemArray: (Recipe) = ()
    
    if withIds.count > 0 {
        
        for recipeId in withIds {
            
            FirebaseReference(.Recipe).document(recipeId).getDocument { (snapshot, error) in
                
                guard let snapshot = snapshot else {
                    completion(itemArray)
                    return
                }
                
                if snapshot.exists {
                    
                    itemArray.append(Recipe(_dictionary: snapshot.data()! as NSDictionary))
                    count += 1
                    
                } else {
                    completion(itemArray)
                }
                
                if count == withIds.count {
                    completion(itemArray)
                }
                
            }
        }
    } else {
        completion(itemArray)
    }

}

}

curated data – Is there a way to get WeatherData consistently?

I can get weather station location:

WMOStations = ResourceObject["WMO Meteorological Stations"];

and select for the US:

cdata = Values@
  ResourceData[WMOStations][Select[#Country == "United States" &]]

Define contiguous states:

kstates = {"AL", "FL", "GA", "MS", "TN", "AR", "LA", "MO", "OK", "TX",
    "AZ", "CA", "NM", "NV", "UT", "OR", "CO", "KS", "NE", "WY", "CT", 
   "MA", "NY", "RI", "DC", "MD", "VA", "DE", "NJ", "PA", "NC", "SC", 
   "IA", "IL", "MN", "SD", "WI", "ID", "MT", "WA", "IN", "KY", "MI", 
   "OH", "WV", "NH", "VT", "ME", "ND"};

Select stations for those states:

data1 = Table[
   cdata[Select[#StateCode == kstates[[i]] &], "Position"], {i , 
    Length[kstates]}];
WMOLocationsUS = Flatten[Normal[data1]];

WMOLocationsUSc = DeleteDuplicates[WMOLocationsUS];

This gives me 397 Weather stations

From these 397 stations, I want to get daily MeanTemperature for the last 10 years. Here’s what I did:

weatherstation = Map[WeatherData, WMOLocationsUSc]; 
meanwdata = 
  Table[{weatherstation[[i]], 
    WeatherData[weatherstation[[i]], 
     "MeanTemperature", {{2010, 1, 1}, {2020, 12, 31}, "Day"}]}, {i, 
    1, Length[weatherstation]}];

The problem is that I get lots of missing values and lots of dates where there is no data. I tried to interpolate the time series and it kinda works but for certain cases, I have to extrapolate and the result is not good.

enter image description here

So, is there a way to filter the weather stations that can give me consistent data? I am fine with have about 300 weather stations.

javascript – Como somar valores de uma lista de objetos agrupando por data “Month”?

Segue exemplo a lista..

  {
    "month": "2021-01-01T00:00:00.000Z",
    "leiteEntregueFaturado": 0,
    "leiteEntregueNaoFaturado": 1000,
    "leiteEntregue": 1000,
    "consumoBezerras": 10,
    "consumoInterno": 40
  },
  {
    "month": "2021-01-01T00:00:00.000Z",
    "leiteEntregueFaturado": 20,
    "leiteEntregueNaoFaturado": 0,
    "leiteEntregue": 20,
    "consumoBezerras": 0,
    "consumoInterno": 0
  },
  {
    "month": "2020-12-01T00:00:00.000Z",
    "leiteEntregueFaturado": 800,
    "leiteEntregueNaoFaturado": 0,
    "leiteEntregue": 800,
    "consumoBezerras": 0,
    "consumoInterno": 0
  },
  {
    "month": "2020-12-01T00:00:00.000Z",
    "leiteEntregueFaturado": 0,
    "leiteEntregueNaoFaturado": 200,
    "leiteEntregue": 200,
    "consumoBezerras": 0,
    "consumoInterno": 0
  },
  {
    "month": "2020-11-01T00:00:00.000Z",
    "leiteEntregueFaturado": 0,
    "leiteEntregueNaoFaturado": 800,
    "leiteEntregue": 800,
    "consumoBezerras": 0,
    "consumoInterno": 0
  }
]

seo – Will Google use field or lab data for ranking sites with core web vitals performance metrics?

Stack Exchange Network


Stack Exchange network consists of 176 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange